Strategies for building Extension-APIs

When writing extension-APIs, the designer usually has two general choices:

  1. The plugin approach: The system can provide a plugin point where external code takes over control and (re)configures the system
  2. The declarative approach: The system calls an user-implementation which computes a token. The caller then incorporates the computed changes into the system.

Both paths have its advantages and disadvantages.

The plugin-path is easy to write, as you literally open all gates and let the foreign code in to refurnish your house. As API designer, your task is simply to define the entry point, and let the user of this API decide what and how to do.

However, it also creates a huge technical dept, as suddenly the structure and behavior of every object and method that is reachable via this API becomes part of the public API. If your internal API is not well-shielded, then you might find yourself in a situation where the user changes parts of the API that you never wanted to open up. The whole situation gets worse as soon as your evil plugin-developers starts to play around with rouge-casts and reflection-APIs.

The declarative approach is more costly to define – as extension-API developer you have to provide a suitable (and hopefully easy to use and understand) model that allows the API-user to fully express the changes he wants to see in the model.

The main advantage of this approach is, that you do not have to open up the private parts of the system. The foreign code runs in its own (more or less protected) sandbox and does not gain control or information about non-related parts of your application.

This limitation of scope makes the whole extension-API easy to understand and easy to use.

The style-expressions introduced in version 0.8.9 are a good example of a declarative plugin-API. The expressions itself compute a value, but it is up to the reporting engine to interpret that value and to make use of it. The engine is free to ignore the computed value, if the value looks like garbage.

The declarative approach is easy for simple property changes, but can evolve into your private version of the 8th circle of hell for large scale tasks. The whole declarative model breaks down as soon as the task cannot be contained in simple data-structures or if the task is somewhat irregular in its nature.

In the very beginning, I used the plugin-approach in the reporting engine. Adding a plugin-interface is easy, and if you make it generic enough, it is a powerful thing. The Functions-API and the old LayoutManager-API (in 0.8.7 and earlier) were examples of a plugin-extension.

The Functions-API is one of the more successful ones. In the Classic-Engine, functions can do pretty much anything they want. They have access to the report-states, the report-definition and all the data-sources. By declaration, functions are allowed to change the report-definition on the fly (which gave us unchallenged flexibility in the report processing) and can do highly complex (and sometimes dangerous) computations.

The LayoutManager-API along with the old output-target-API was a example of an API where the plugin-approach backfired. The idea behind that part was to make layouting and content-generation more extensible. For this part, the inital design process was the equivalent of dropping code on a white canvas to see what sticks. Over time, the implementations became insane in itself – with workarounds for workarounds for bugs.

As the initial API was very generic, there was no controlled way to fix the issues we had with the underlying objects. Any fix to the (formerly) private objects now threatened to break the client code that used that API. There is only one way to fix such an API: To burn it down, salt the earth on which it grew and start from scratch (which we did in version 0.8.9).

The Functions-API had similiar problems, but due to the way that API was received by its users, these problems never evolved to the same level of pain as the layouting-API. What might have saved us here, was the fact that Functions and Expressions were always seen as tied to a certain purpose – compute values or style report-elements.

During the last few years, I learned to value the declarative approach. With a declarative API, you invest time to protect the user from shooting himself in the foot. Although foot-shooting can be a healthy experience (as it can reinforce values like “think-before-you-code”, careful planing or clean and maintainable implementations after you shot yourself hard enough), most of the time the costs still do not justify the outcome. As architect of the outside code, I have to spend hours and hours on protecting the precious parts of the system against evil code. I have to write large documents to spell out in english words what you should do and more often what you should not do with the API. With every change of the internal API I spend hours to think about where and how this change now breaks code in some remote plugin. At the end, some code always breaks.

With the declarative approach, there is only one simple (and hopefully unbreakable) interface. The foreign code is perfectly shielded from the inner details of the engine. The code itself cannot mess with internals of the engine and cannot cause invalid states. Even if the code returns garbage, the engine has its chance to detect this and either fix or reject the request.

There will be only few cases in the life of an software engineer, where a pure declarative approach would be feasible. I also would not go so far to abandon the use of the plugin-approach altogether, but as with the use of nuclear weapons – if you use a pure plugin-API-approach you better have good reasons. Your actions may cause more trouble than you initially bargained for…

This entry was posted in Architecture on by .
Thomas

About Thomas

After working as all-hands guy and lead developer on Pentaho Reporting for over an decade, I have learned a thing or two about report generation, layouting and general BI practices. I have witnessed the remarkable growth of Pentaho Reporting from a small niche product to a enterprise class Business Intelligence product. This blog documents my own perspective on Pentaho Reporting's development process and our our steps towards upcoming releases.