One Single Version of the Truth does not exist

This week I went to a seminar at Logica called Guru4Pro. The subject was: “Best Practices in Data Vault”, and the Godfather of Data Vault, Dan Linstedt, was one of the speakers.
There were also two other speakers, who each explained a real-world example of a Data Vault implementation. They were Aytekin Keskin, who worked as an architect on a Data Vault project at Tele2, and Liesbeth Hozee, who helped with a Data Vault project at Vektis.

After the introduction the speakers put a lot of emphasis on the challenges the projects ran into. What stood out was that these were mainly technical. At Vektis, for example, recognizing good source keys when different parties deliver patient data (how do I recognize a unique patient then?). At Tele2 the challenge was the huge number of requirements, more than 3000, and translating them into a small number of functional areas.

What I always like about talks like these is that they make you think. At one point Dan Linstedt said: “A single version of the truth doesn’t exist”. But then what have we been doing all these years with our data warehouses? Exactly, creating the truth. But what is the truth, and who decides what the truth is? Dan gave a nice example that sticks. He talked about a lamp with a yellow bulb in it. If someone else puts in a blue bulb, you get a different colour of light. Then someone else comes along and puts in a green bulb. The bulb was the truth, and changing the bulb was the business rules. What he tried to say is that the business decides what the truth is. As soon as the business changes, or new managers come in who benefit from a different truth, the truth will change. A truth in a data warehouse is a set of rules at a certain moment for a certain person.

Data Vault handles this well by keeping raw data: the data as it is stored in the various source systems. The truth is as it was stored in the systems, without changing any data. By changing I mean cleaning and transforming. Because of this the data is always 100% auditable/traceable. Someone also asked whether you are allowed to convert, for example, a date field stored as text to a date field. That question was left open.

To then turn data into the truth, the “Version of the truth”, Dan says you should use Business Vaults in the broadest sense of the word. This does not have to be a physical data mart such as a Kimball model, an OLAP cube or a data vault with cleaned data. Business Vault is a very broad concept. It covers basically everything that is built on the data vault and that turns the data into meaningful business data. This can also be an Excel sheet with business rules (with PowerPivot, for example), but also a physical database modelled according to the Kimball method.