A central place for all your data

Make data a key factor within your organisation

Datawarehousing

Data warehouse automation

For a company having a data warehous is not a stand alone goal, but a resource. You want insights in your operations by using reports and dashboards, you are not looking for a data warehouse. That is why we have chose to use Biml. Biml stands for Business Intelligence Markup Language, and can be used to generate datawarehouses with. The advantages are:

Quick

En therefore lower costs.

Good data quality

It is possible to check the data quality centrally.

Version management

Keep track of the changes of the code that the data warehouse generates.

Flexibility

Biml makes it possible to make changes once, like the adding of logging, and implement it everywhere.

Uniformity

With Biml it is easy for a team to use the same practice.

More insights

It is esay to document and make changes, therefor it is easier to understand the operation of the data warehouse.

For Microsoft Dynamics AX we have a specific data warehouse accelerator, so you have your first reports and dashboards real soon. Do you want the advantages of a data warehouse, but only invest in the activities that add value? Call us for more information.

On the right the business intelligence maturity model of Gartner is presented. All models are a simplistic view of reality, but this model is very useful for picturing where you are at the moment, and where you want to be in the future. A data warehouse is mostly implemented around level 3, ‘focused’. At this level you have issues like:

  • My reports are slow and/or my ERP systems gets slow when users execute heavy reports
  • Our controllers are most often busy reporting, while I would like to see them analysing
  • I want to combine data from more than one source in a report
  • I have the feeling that I repeate certain actions for different reports (like filtering data)
  • There are differences between reports that should have the same results
  • My source systems saves not enough historical data
  • We have too many Excel sheets to keep track/When our Excel expert stops, we have a big problem
  • Managers want to have direct insights, instead of waiting for the monthly report
  • We want more control over who has access to which data
  • Our data should be structured and cleaned before reports can be build
gartner business intelligence maturity model
If you recognise one or more of these issus, it may be interesting for you and your organisation to look into the possibilities of a data warehouse.

Data quality in the data warehouse

When building a data warehouse, we can directly take data quality into account. Ofcourse, all data is cleaned first before loading it into the data warehouse. Moreover, it is possible to keep track of metadata (data over data). Possibilities are:

  • Where does this data come from?
  • Which transformations are done while loading the data?
  • Is this a possible outlier (very low/high)?
  • Which version of the data warehous loaded this record?

By adding these kinds of metadata, it is possible to add information to reports like ‘4% of the records used for this report may be of lesser quality, click here to look into the issues’. This makes the management aware of data quality and makes it possibile to keep it in mind while making decisions.

Data Vault

Data warehouses based on the Kimball methodology are usefull for smaller companies with limited amounts of data. In complex businesses, there are 5 issues with which business intelligence teams have to deal with during the implementation and maintenance of a Kimball data warehouse. Data Vault can handle these issues.

Kimball (old) vs Data Vault (new)

  1. In the case of big data, the cleaning en processing of data can take a long time. Data Vaults saves all the data in a raw format in a Raw Data Vault. The time needed to process is therefore quicker, since there is no cleaning process in between.
  2. “The one constant is change”. Companies change their processes and their business all the time. Because of this, new data enters the data warehouse reguarly. A Data Vault model is easier to adjust to the new situation.
  3. Complexity. Data Vault makes the design easier and therefor avoids complexity. Because of this it is possible to better adjust tot the wishes of the customer, in stead of focussing on the technical problems.
  4. Data Vaults are easier to generate automatically then a Kimball model
  5. Tracking history is difficult in a Kimball model, certainly when new data enters and calculations have to be done retroactively. In Data Vault data is saved with a timestamp, so you can easily track changes over time.
  6. The latest version of Data Vault, version 2.0, makes it even more easy to deal with different kinds of data that are saved in the present time (not structured, NoSQL databases, etc).

Kimball and Data Vault compared visually

Data Vault vs Kimball

Data Vault (left) en Kimball (right) source: Supercharge your datawarehouse (Linstedt)

On the left Data Vault is represented, consisting of links, hubs and satellites. On the right the Kimball model is pictured, consisting of facts and dimensions. When you translate the DataVault model to the Kimball model, then there would be a fact table fact_Order with measures like amount, and links to the dimension tables. Both dim_Product and dim_Customer would be a dimension table. With this you can analyse the total amount of orders per customer and product. Moreover, you can summarize it to higher levels, like the total amount of orders per product group. Because this is easy to work with for analysts, it is ofter seen that the Kimball model is used upon the Data Vault. In this way the Data Vault can focus on recording historical data the right way and the Kimball model is used to answer business cases.

A typical Data Vault architecture

The German company Dörffler & Partner, that Data Vault expert Michael Olschimke worked for, has pictured the typical Data Vault architecture:

Data Vault schema

The architecture consists of the following steps (between parentheses the naming):

  1. Copying of data from the different source systems to the staging area
  2. Loading of the Raw Vault, that contains 100% of the data from the source systems and where optionally data from an Enterprise Service Bus (like BizTalk) can be added.
  3. Thereafter, the Business Vault is loaded, where business rules are applied. Data is cleaned, bad records are deleted, etc. This means that the Data Vault does not necessarily contains 100% of the data.
  4. The Business Vault serves as source for:
    • A Kimball model (star schema)
    • Raw data marts (raw data, these marts contain 100% of the data, also the bad data)
    • Reports
    • Error data marts: cointain the bad data, this data is useful for employees who are responsible for data quality

When a Data Vault?

Compared to a lot of other business intelligence consultants, we do not believe blindly in Data Vault. We even stopped proceding with Data Vault once, when we noticed that the customer had implemented it for the wrong reasons. We only see the added value for your company when you:

  • work in a fast changing environment
  • work with big data
  • have to justify how certain figures are created (auditability)
  • have to report on historic changes

For more information about a Data Vault implementation, or when you waver between Data Vault and other methods, please contact us.

Interested?
Request a free demo!

We will reach out to you within one day!

Request Demo

Contact

Interested in one of our products or our company? Fill in the following form or send us an email info@dynamicinfo.nl. We will reach out to you within one day!

Dynamic Info

Building ’Koningshof’
Schipluidenlaan 4
1062 HE Amsterdam

Tel. +31 (0) 203 032 470
Fax. +31 (0) 848 388 100
E-mail. info@dynamicinfo.nl

Chamber of Commerce: 65061284
VAT-number: NL855965265B01