Design Patterns in Data Migration

Design Patterns in Data Migration
This page starts with the suggestion that Data Models are 'A Good Thing' and you can never have too much of 'A Good Thing'
Therefore, if the answer to everything is a Data Model then what exactly are the Questions ?

We propose that an End-to-End range of Data Models provides a robust and generalised foundation for any Data Migration activity.
Of course, we consider it is true that 'Data Migration' is a deceptively simple way of describing the activities that occupy most of our waking (or working) hours most of the time.
This is appropriate whether we are Data Architects, Data Modellers or Data Analysts, where each role has a different range of skills and responbsibilities.
We start with Operational Data Stores and end with a Dimensional Model that feeds data for a BI Layer.

Our question then becomes :-
What kind of Data Models do we need to cover our End-to-End Scope ?

We propose a range from ODS, Data Mapping, Data Warehouse, Dimensional Models and Semantic Models

This seems a modest ambition, so let's discuss it in detail to see if it holds up under detailed analysis.

This page shows four Design Patterns in Data Migration.

1) The first Design Pattern shows Semantic Data Models which provide a 'User Friendly' front-end.
This helps Users to request KPIs, Reports and analyses using words and terminology that they normally use.
For example, talking about Customers, rather than Parties.
If Semantic Data Models have been created, then we can say that the work has been done to the established standards of Best Practice.

2) The second Design Pattern shows Mapping Specifications which translate Source data to Target data.

3) The third Design Pattern identifies KPIs, and traces their derivation.
This helps to establish the Data Lineage for Sarbanes-Oxley and other Statutory Reporting requirements.

KPIs, Data Lineage and Agile Modelling

4) The fourth Design Pattern shows the complete End-to-End scope of Data Models.
The Common Data Model is used to standardise data that is loaded into the Data Warehouse (DWH).
We can also be sure that we have a 'Single View of the Truth' at any point in time.
It helps us to build the design of the DWH in a step-by-step fashion.

We identify that the Enterprise Data Warehouse is a top-level design which will have lower-level Subject Area Models such as Customer Purchases.
End-to-End Data Model Foundation

Finally, we show the complete Data Warehouse Bus Architecture