Data management is the management of data from beginning to end (archiving/deleting). A datawarehouse is an important part of this management, because it can integrate, maintain and archive data from different sources simultaniously. The buying of new applications, merges and acquisitions and other causes make it necessary that data can be migrated easily to other systems so operational processes are not disturbed. Moreover, it is important that through the lifespan of data the quality is ensured.

The BI consultant helps with the implementation of your source system

Quite often a business intelligence consultant is hired after a source system is implemented. By this time budgets are exceeded and only few resources are left for the business intelligence project. Morover, during the project it is found out that the data is implemented well for the operational process, but not for reporting.

Therefore we highly recommend to involve a business intelligence consultent from the beginning of the data lifecycle. The beginning is where the data is created: the implementation of a new application. By working closely with our colleagues from Dynamic People, we are able to this for their applications, but we can also help with other applications. It is possible to save many hours of solving technical challanges by having a few hours of consultancy at the beginning of the process. Moreover, a well designed and maintained application prevents quality problems with data, like wrong input in fields.

Would you like our opinion on your data lifecycle management? Do not hessitate to contact us for more information.

Data migration

With the migration of data from an old to a new system it is important to map the fields from the old system on the new one, and convert them where necessary. SQL Server Integration Services is a good tool to make a mapping and repeat the process untill it is perfected. It is also possible to generate automated mappings, which saves a lot of manual work. Even better, these mappings can be automatically read which makes the documentation of the migration process easy.

Mapping between input and destination columns in SSIS data migration

Data migration by Dynamic Info

There are several reasons why it is advantageous to let Dynamic Info migrate your data:

• Short turnarounds by automatically reading tables from the old and the new systems, and the automatic mapping of big parts of columns
• Automatic conversion (data types, splits/merges, etc.)
• Automatic generation of migration documentation
• Replicable migration steps

Convinced of the benefits? Contact us to discuss your migration wishes.

Data quality

Data of high quality is one of the biggest success factors for business intelligence. ‘Garbage in, garbage out’ (GIGO), or: bad data almost always means bad reporting. There are limits to improving data quality afterwards (while loading), so it is important that the quality is good.

Organisation

Unfortunally, data quality is neglected by companies. It has rarely the attention of the higher management, while the consequences of bad data are felt in the management and profitability of the company. At one of our customers we have embedded the data quality and problems in the balance scorecard of every team. In this way it was possible to see which departments were doing well. Moreover, it became apparent which teams where actively trying to improve their data quality. Also with the subject of data quality, the ‘tone of the top’ is important, like making teams responsible for their own quality of data (data stewardship).

Technical help

To obtain better data quality, it is possible to use technical tools. Mostly these are standalone applications, but there are also tools which are part of the data warehouse.

BI in the Cloud

Business intelligence in the cloud is coming soon. The advantage of the cloud is the distinction between compute and storage:

• The refreshing of the data warehouse and the execution of an analysis are both compute. When nothing happens (for example at night), then you only pay for storage
• For compute you pay by the hour, which varies (at the moment) between €0.66 and €13.22 an hour, depending on the power you need. This means flexability to choose to refresh or compute more during peak hours

Source systems like ERP- and Microsoft Dynamics CRM systems move increasingly to the cloud. It is easier to compare these data in the cloud for analysis en reporting (in stead of moving the data to your own servers). From 2015 Microsoft provides SQL Data warehouse in the cloud. This works seamlessly together with the other Microsoft cloud applications, like:

• Power BI: fast insights in your data, using dashboards on the computer, tablet and smartphone
• Azure Machine Learning: Training models to better understand and forecast data, based on historical data. The possibilities are endless, and vary from the segmentation of customers, detection of fraud, estimate credit risks, better forecasting by recognising hidden patterns, etc.
• HD Insight: using Hadoop, Spark and R in the cloud. Hadoop is almost a synonim for big data, and is used to quickly save data whithout thinking about a (database) structure. The data is devided over different dervers. On the downside, loading data for analyses are quite slow. Spark solved part of this slowness by, for example, using shared memory for the servers. R is a advanced, open source statistical tool with a highly active community, that created many extensions.
• Data Factory: loading and combining data from different sources within the cloud