A Methodology for Low-Code Business Automation with BPMN and DMN

In recent posts, I have explained why anyone who can create rich spreadsheet models using Excel formulas can learn to turn those into Low-Code Business Automation services on the Trisotech platform, using BPMN and DMN, and why FEEL and boxed expressions are actually more business-friendly than PowerFX, the new name for Excel’s formula language. That’s the why. Now let’s discuss the how.

In recent client engagements, I’ve noticed a consistent pattern: A subject-matter expert has a great new product idea, which he or she can demonstrate in Excel. A business event, represented by a new row in Excel Table 1, generates some outcome represented by new or updated rows in Tables 2, 3, and so on. The unique IP is encapsulated in the business logic of that outcome. The challenge is to take those Excel examples and generalize the logic using BPMN and DMN, and replace Excel with a real database. This constitutes a Business Automation Service.

Business Automation Service

The Basic Pattern

In BPMN terms, a Business Automation service typically looks like this:

Each service is modeled as a short-running BPMN process, composed of Service tasks, Decision tasks, and Call Activities representing other short-running processes. The service is triggered by a business event, represented by a client API call. The first step is validating the business event. This typically involves some decision logic on the business event and some retrieved existing data. If valid, one or more Service tasks retrieve additional database records, followed by a Decision task that implements the subject matter expert’s business logic in DMN. In this methodology, DMN is used for all modeler-defined business logic, not just that related to decisions. The result of that business logic is saved in new or updated database table records, and selected portions of it may be returned in the service response. It’s a simple pattern, but fits a wide variety of circumstances, from enterprise applications to TurboTax on the desktop.

Implementation Methodology

Over the past year, I’ve developed a methodology for implementing that pattern.

It goes like this:

The first step is whiteboarding the business logic in Excel. In many cases, the subject matter expert has already done this! It’s important to create a full set of example business events, and to use Excel formulas – with references to cells in the event record – instead of hard-coded values. If you can do that, you can learn to do the rest. The subject matter expert must be able to verify the correctness of the outcome values for each business event.

Now, referring to the BPMN diagram, we start in the middle, with the Decision task labeled here Calculate new/updated table records. A BPMN Decision task – the spec uses the archaic term businessRule task – invokes a DMN decision service. So following whiteboarding, the next step in the methodology is creating a DMN model that generalizes the whiteboard business logic. I prefer to put all the business logic in a single DMN model with multiple output decisions in the resulting decision service, but some clients prefer to break it up into multiple Decision tasks each invoking a simpler decision model. That’s a little more work but easier for some stakeholders to understand. This DMN model is in one sense the hardest part of the project, although debugging DMN is today a lot easier than debugging BPMN, since you can test the DMN model within the modeling environment. The subject matter expert must confirm that the DMN model matches the whiteboard result for all test cases.

On the Trisotech platform, Decision tasks are synchronized to their target DMN service, so that any changes in the DMN logic are automatically reflected in the BPMN, without the need to recompile and deploy the decision service. That’s a big timesaver. Decision tasks also automatically import to the process all FEEL datatypes used in the DMN model, another major convenience.

Next we go back to the beginning of the process and configure each step. The key difference between executable BPMN models and the descriptive models we use in BPMN Method and Style is the data flow. Process variables, depicted in the BPMN diagram as data objects, are mapped to and from the inputs and outputs of the various tasks using dotted connectors called data associations. Trisotech had the brilliant idea of borrowing FEEL and boxed expressions from DMN to business-enable BPMN. In particular, the data mappings just mentioned are modeled as FEEL boxed expressions, sometimes a simple literal expression but possibly a context or decision table. If you know DMN, executable BPMN becomes straightforward. FEEL is also used in gateway logic. Unlike Method and Style, where the gate labels express the gateway logic, in executable BPMN they are just annotations. The actual gateway logic is boolean FEEL expression attached to each gate.

Database operations and interactions with external data use Service tasks. In Trisotech, a Service task executes a REST service operation, mapping process data to the operation input parameters and then mapping the service output to other process variables, again using FEEL and boxed expressions. The BPMN model’s Operation Library is a catalogue of service operations available to Service tasks. Entries in this catalogue come either from importing OpenAPI files and OData files obtained from the service provider or manually creating them from service provider documentation. It sounds difficult, but it’s actually straightforward, and business users can learn to do it.

Example

A Stock Trading App

Here is an example to illustrate the methodology. Suppose we want to create a Stock Trading App that allows users to buy and sell stocks, and maintains the portfolio value and performance using three database tables: Trade, a record of each Buy or Sell transaction; Position Balance, a record of the net open and closed position on each stock traded plus the Cash balance; and Portfolio, a summary of the Position Balance table. In Excel, a subject matter expert models those three tables. Each trade adds a row to the Trade table, which in turn adds two rows to the Position Balance table – one for the traded stock and one for Cash, with total account value and performance summarized in the Portfolio table. The only part of this that requires subject matter expertise is calculating the profit or loss when a stock is sold. When that is captured in the Excel whiteboard, translating that to DMN becomes straightforward. The BPMN Decision task executing this DMN service has the added benefit of automatically including the FEEL datatypes of the three tables in the process model.

Next we need to address data validation of a proposed trade. In a previous post, I explained how to do this in DMN. We need to ensure that all of the required data elements are present, and that they are of the proper type and allowed value. If we do not allow trading on margin or shorting the stock, we need to retrieve the account Portfolio record and make sure there is sufficient Cash to cover a buy trade or sufficient shares in the Portfolio to cover a sell trade. A full-featured trading app requires streaming real-time stock quotes, but in a simplified version you could use real-time stock quotes on demand via the Yahoo Finance API, which is free. As in the case of many public APIs, using it requires manually creating the Operation Library entry from service provider documentation, but this is not difficult. Any problems detected generate an error message, and a gateway prevents the trade from continuing.

Also as explained in previous posts, we can use the OData standard to provide Low-Code APIs to the database tables. OData relies on a data gateway that translates between the API call and the native protocol of each database, exposing all basic database operations as cloud REST calls. In its OData integration, Trisotech does not provide this gateway, but we can use a commercial product for that, Skyvia Connect. The Skyvia endpoint exposes Create, Retrieve, Update, and Delete operations on the three database tables, and Trisotech lets us map process variables to and from the operation parameters using boxed expressions and FEEL.

While this Low-Code methodology is straightforward, to become adept at it requires practice. Executable BPMN, even though it’s Low-Code, requires getting every detail right in order to avoid runtime errors. Debugging executable BPMN is an order of magnitude harder than DMN, because you need to deploy the process in order to test it, and instead of friendly validation errors such as you get in the DMN Modeler, you usually are confronted with a cryptic runtime error message. (Note: Trisotech is currently working on making this much easier.) So the modeler still must spend time on testing, debugging, and data validation up front – things developers take for granted but try the patience of most business users.

Using this methodology, it is possible for non-programmers to create a working cloud-based database app based on BPMN and DMN, something most people would consider impossible.

Learning the Methodology

It’s one thing to read about the methodology, quite another to become proficient at it. It takes attention to the small details – many of which are not obvious from the spec or tool documentation – and repeated practice. To that end, I have developed a new course Low-Code Business Automation with BPMN and DMN, where each student builds their own copy of the above-mentioned Stock Trading App. Students who successfully build the app and make trades with it are recognized with certification. Students need to know DMN already, including FEEL and boxed expressions, but the course teaches all the rest: data flow and mapping, Operation Library, OpenAPI and OData, configuration of the various BPMN task types for execution.

The course is now in beta. I expect some enhancements to the Trisotech platform that will be incorporated in the GA version around the end of the year. In the meantime, you can take a look at the Introduction and Overview here. Please contact me if you are interested in this course or have questions about it.