A bit of testing never hurts ..

Each time I touch the layouting subsystem of the reporting engine, I am scared to death. Layouting is a task that is not simple and almost never straight forward. Unless you are willing to consume loads of memory (up to the point where you would have to invent swapping system to free memory for larger reports) the layouting system needs to be balanced carefully.

The layout rules must be deterministic, complete and at the same time easy enough to understand to tell them to the user.

Our layouting engine is basically a huge state machine, where nodes get added and removed constantly to represent both an accurate picture of what has been processed (so that space is not allocated twice) and what is currently due to be processed. The engine itself streams data though its guts and only holds on to the parts that have not yet been printed. It allows us to have a really minimalistic memory footprint, nearly regardless of the total number of pages rendered. One page or a million, the amount of memory used stays a rather flat line.

On the negative side, we sit on a huge pile of states, all interdependent and interwoven and nearly impossible to simulate from outside. Some parts can be extracted, but the majority of the code needs to be run together to behave correctly.

Writing test cases for that is not fun – until now.

Over the last few weeks I wrote a set of “golden sample” tests. A golden sample is a test that runs a given report and compares its output against a known good state. Such tests cannot provide protection from unknown evils, but we are now at least in a position to validate that the existing reports run as before.

And best of all: These tests are easy to set up and easy to maintain. As long as we uphold the promise that old reports behave the same in newer reporting engines, we can just drop the report definitions into the sample pool, generate golden outputs and let them stay there for future generations of testers.

The test system could easily used outside of the development cycle as well. If you let your reports run against a test database you could verify whether the latest upgrade broke your reports or whether test system and production system *really* produce the same results. What do you think – would a “golden test” make sense in your organization?

This entry was posted in Development on by .

About Thomas

After working as all-hands guy and lead developer on Pentaho Reporting for over an decade, I have learned a thing or two about report generation, layouting and general BI practices. I have witnessed the remarkable growth of Pentaho Reporting from a small niche product to a enterprise class Business Intelligence product. This blog documents my own perspective on Pentaho Reporting's development process and our our steps towards upcoming releases.

2 thoughts on “A bit of testing never hurts ..

  1. gunter.rombauts

    Yes a “golden test” would mean that we can upgrade to a newer version faster and with less risks. Currently the testing effort involved, makes us decide to skip a newer version. Then we ask you to backport …..

    When will it be available to get my hands on it?

  2. Thomas Morgner

    I should have a download around this time next week. As we are using the XML output (and due to the fact that this one changed a bit in 4.0 to make these tests possible [filter out changing attributes, fix the report.date to a static value etc]), these tests will only make sense from 4.0 onwards.

    But then again, there is no way you want to skip 4.0 and crosstabs anyway 🙂

Comments are closed.