DMN: Dealing with Nothing

They used to say Seinfeld was a show about Nothing. But given its enduring influence on popular culture, Nothing is clearly not nothing. Attention must be paid.

In DMN, as well, Nothing demands attention. In simple models like the ones we study in my DMN Method and Style Basics and Advanced training, it doesn’t come up. But in many real-world decision models, Nothing pops up all over the place, often unexpectedly. When it does, proper attention must be paid or you will not get the correct result. This post shows you how.

A typical example is a model used to validate data prior to making the decision. FEEL is a fine language for data validation. Its functions and operators are rich enough, and it makes the logic transparent and business-friendly. But it is a little clumsy when it comes to handling Nothing, which in FEEL comes in two forms: null and an empty list [ ].

In real DMN models, it is often the case that some input data is missing. That’s not a mistake. In XML, those elements would be designated as optional in the schema, but FEEL does not have this concept. In the DMN input data, there is no distinction between required and optional elements, but in practice the optional elements often are missing. If a simple type, the missing element is assigned the value null; if a collection (list type), it is assigned the value [ ], i.e. an empty list; if a structured type, well let’s defer that one for now.

A common case of a missing simple type is a form field left blank.  For example, I’ve written before about the new standard Uniform Residential Loan Application (URLA) form for home mortgages.  Below is a clip of that form detailing the borrower’s monthly employment income:

Suppose you want to validate that the Total Income field matches the sum of the income items.  To do that, you could try some variant of the logic below:

Income Items: [Base, Overtime, Bonus, Commission, Military Entitlements, Other]
		Total Income = sum(Income Items)

But that test would fail.  Because some items have the value null, the sum is also null.  A better result would come from using

Total Income = sum(Income Items[item != null])

This filter removes the blank values from the list, and in this case the test now works.

But that’s not the end of the story.  This part of URLA is just wage income; it does not include self-employment income or other income.  So suppose all the Income Item elements are blank and the Total Income field is 0.  The test now fails!  The argument of the sum is an empty list, and for some reason (not a good one), the spec says

sum([])=null

That suggests the test you want is

Total Income = if Income Items[item != null] = [] then 0 else sum(Income Items[item != null]

Wow, that’s a mouthful!  Because this comes up all the time, it’s better to make this a BKM:

nn sum(numList):  if numList[item != null] = [[] then 0 else sum(numList[item != null])
		Total Income = nn sum(Income Items)

Actually, Trisotech and Red Hat support nn sum and several other nn (non-null) functions as built-in extensions, so you don’t have to create the BKM yourself.

And I need to mention that empty lists don’t just arise from blank fields on a form.  They occur all the time when a filter returns no list items, and this is very common.  That means just as with null, any time you are dealing with lists you need to account for the possibility of an empty list.  If you have an expression like

some x in myList satisfies ...

you could get a runtime error if myList=[ ].

Turning back to our URLA example, what if all the Income Item elements are blank and Total Income is blank as well?  In that case, the test above still fails!  When a blank numeric element is interpreted as 0, you need a function to map it to 0.

num0(myNum): if myNum=null then 0 else myNum

So at last we have a reliable test for Total Income:

num0(Total Income) = nn sum(Income Items)

As I said, when your DMN handles Nothing, attention must be paid!

A common feature of data validation models are error messages, and Nothing must be accounted for here as well.  Typically these error messages concatenate some fixed test and values from the model.  For example, suppose the Total Income value from URLA must match a corresponding value on some other form, such as Form XYZ.Total Monthly Income.  The error message would be something like this:

Error Message: "The Total Monthly Income " + string(FormXYZ.Total Monthly Income) + " reported on Form XYZ does not match the value " + string(Total Income) + " reported on URLA."

We need the function string() to cast a number to text for use in the concatenation.  But what if Total Income on URLA is blank, and FormXYZ.Total Monthly Income is some non-zero value?  As you probably have guessed,

string(null) = null

and so

Error Message = null

That’s not what we want, obviously.  Here we have a choice.  We could define Error Message as

"The Total Monthly Income " + string(num0(FormXYZ.Total Monthly Income)) + " reported on Form XYZ does not match the value " + string(num0(Total Income)) + " reported on URLA."

or as

"The Total Monthly Income " + stringn(FormXYZ.Total Monthly Income) + " reported on Form XYZ does not match the value " + stringn(Total Income) + " reported on URLA."

where the BKM stringn is defined as

stringn(myNum): if myNum=null then "null" else string(myNum)

The first one reports null values as “0” and the second reports null values as “null”.  Either one is ok.

Is your head spinning yet?  OK, one more.  I mentioned at the top that if an input data element is a structure and missing entirely on execution, its value is… what?  That’s a tough one.  In XML it’s easy, since there Nothing is represented by exactly that, i.e., nothing.  It’s not included in the instance at all.  Let’s say Form XYZ is missing entirely in the data.  You can test for that in XPath using the Boolean functions exists(FormXYZ) or not(FormXYZ).  But it’s trickier in DMN, and partially tool-dependent.  In the DMN runtime, you can use the expression Form XYZ=null and it works just like the XML case.  But in a tool like Trisotech Decision Modeler, leaving Form XYZ blank in the Execution/Test html form does not work, because it interprets Form XYZ as a structure containing null components, so Form XYZ = null returns false.  That’s a valid interpretation as well.  This all comes from the fact that FEEL does not support optional elements, and the spec is silent about their value when the data is missing.  To solve this issue, the safest thing is to test a component of the structure that will always be present if the element is not missing, such as id.  To test if Form XYZ is missing, you should

use something like

Form XYZ.id = null and...

In spite of these quirks, DMN is actually a great language for data validation models.  The logic is transparent and understandable to business stakeholders.  And once the data is valid, it can be used within the same decision model to evaluate the rest of the decision logic.  This is much better than validating the data first in Java and then passing clean data to DMN.