Article by COSOL (Data Team)
Our Point of View
25+ years of experience shows that one critical success factor for these programs to realise the value in their business cases is Data Migration. The axiom "garbage in, garbage out" remains as relevant today as it did when computers were first invented. COSOL strongly believes:
that digital transformation is well underway, and every Board is, and should be worried about how to become a truly digital enterprise.
strong Enterprise Data foundations will be required to enable adoption of digital solutions including advanced analytics, robotic process automation, machine learning and artificial intelligence which are the next frontiers to productivity and market competitiveness.
Enterprise Data is the glue, the fact base, that drives decision making and business improvement, allowing organisations to meet stakeholder expectations in a timely and efficient manner; and
for organisations to succeed, Data must be treated as a mission critical Asset; it is the single biggest success factor in a digital transformation journey, and most organisations are ill prepared due to many islands of disconnected data that is of unknown and/or poor quality.
Introduction to Data Migration
| Pre-Migration | During Migration | Post-Migration |
|---|---|---|
| Step 1 Data Profiling & Remediation |
Step 2 Data Standardisation & Loading |
Step 3 Data Reconciling & Archiving |
Recapping the Key Takeaways from the Pre-Migration:
- Data will typically be exposed as a major risk and cost in large scale digital transformations due to unknown and/or poor quality.
- Strong Data Ownership and Data Governance is Critical; and
- Commence Data Profiling & Data Remediating as early as possible.
With these insights, this article examines the step During Migration where data is Standardised then Loaded into the target system(s).
As can be shown in Figure 2: Data Standardising and Loading, the Six Dimensions of Data Quality covered in the previous article in this series will continue to be remediated throughout this step. This reinforces the importance of the key takeaway about Strong Data Ownership and Data Governance, and serves as a reminder also from the previous paper that "this Is a business issue first and foremost and that the business must take ownership and accountability of its data as a strategic asset". The project team must have empowered Owners in place to make decisions about their data for the program to succeed.
Another key takeaway in the previous paper suggested Data Owners start Business Remediation early by identifying and deleting Duplicate and Redundant data in existing systems. The programs focus would be on the other four remedies shown in Figure 1: Standardising data formats, correcting Incorrect data values, fixing data Integrity errors, and enriching data for Completeness.
Each of these six remedies need to be completed to prepare data for loading into the target system. By now, the program would be mobilised, and the team would work with, or contain Functional Consultants with expertise in this area to work with the business Data Owners.
Takeaway #1
In all, six remedies need to be completed through this step. A reminder to commence Data Profiling and deleting Duplicate and Redundant data records as soon as possible.
Data Quality Remedies
Standardise Data Formats
This remedy addresses the situation where Existing System A and B records items in different formats to each other, and/or to the format required by the new system.
An example could be Suppliers in System A use alpha characters (e.g. aaa) and System B uses numeric characters (e.g. 000). The new system possibly uses a different standard again (e.g. aaa000). Such data maps need to be developed for each data object, approved by the Data Owner and then used by the project team to convert existing system formats to the new system format.
Correct Data Values
This remedy is to ensure that the Values contained within existing systems in the old format, successfully migrate to the new system and in the new format. Using the Data Format example above, Suppler A might a value in System A of XYZ (alpha) and 001 (numeric) in System B.
In the new system Supplier A has been designated XYZ001 (alpha-numeric) and hence XYZ from System A must map to XYZ001, as must 001 from System B. Another simpler example might be either misspelling (suplier vs. supplier) and/or completeness (Supplier vs. Supplier Pty Ltd), all of which will need correcting when encountered.
Fix Data Integrity
It is entirely possible that as new data formats and new data values are developed, that some data records become orphaned. e.g. A purchase order is no longer associated with a Supplier. This should be caught as a violation through the migration testing and corrected before Data Loading.
Enrich Data Fields
This remedy addresses two common situations. One situation is simply where some existing data fields are incomplete (e.g. some addresses are missing the post code).
The other situation is where the new system requires new information not previously captured nor stored in existing systems. In both situations, effort will be required to manually find the missing data and make it available for Data Loading into the new system.
Data Standardising
The actual process of applying these remedies to Standardise the data for the new system is an iterative process consisting of multiple "Mock Runs". These can simply be thought of as dress rehearsals for the final cutover to the new system, and like any rehearsal, the outcomes of each improve with practice.
After each mock run, the Data Owner would receive a view of their data quality along with a list of ‘violations'.
Decisions would be made and actions taken to address the violations, with the action being one of the remedies previously described:
- delete the records as duplicate or redundant,
- update the data map to cater for an offending format,
- update the value map,
- fix an integrity issue and/or find the missing data.
Takeaway #2
The actual process of applying these remedies to Standardise the data for the new system is an iterative process consisting of multiple "Mock Runs".
| Mock 0 | Objective(s) | Identify as many quality and performance issues as possible to inform project plan |
| Target | 100% of key Load Programs run with as large a dataset as possible | |
| Mock 1 | Objective(s) | Successfully prove key Load Programs and Test mapping for key objects on limited data |
| Target | 100% of key Load Programs successful | 10% of data loaded successfully | |
| Mock 2 | Objective(s) | Test key data mappings, refine key relationships and define pre & post validation criteria |
| Target | 50% of data loaded successfully | |
| Mock 3 | Objective(s) | Go live dress rehearsal and confirm cutover timings and load sequencing |
| Target | 100% of data loaded successfully |
Data Loading
The Data Migration stream within the program has reached a critical milestone where the six remedies have successfully been applied and tested.
Data Owners and the project team can confidently migrate the relevant data from existing systems in existing formats with existing values, to the new system(s) in the new formats complete with new values, in the right sequence, per the system design. Attention now turns to the actual Data Loading and the integrated cutover plan.
The Integrated Cutover Plan
An integrated cutover plan is critical to the overall success of the program as it minimises the risks, the time,and the possible business disruption of cutover.
Planned and executed well, an integrated cutover plan will drastically reduce the amount of reconciliation and manual tasks required by the business post go-live. It will inform the key requirements for the Reconciliation process. A simple example is if all invoices are approved and paid before cutover, then there is no need to reconcile the accounts payable open items.
Similarly, it is also important to plan and agree the data migration approach for sub-system loads (e.g. Stock Balances) as this will result in not having to migrate and reconcile the general ledger account balances for Inventory.
Examples of Best Planning in Cutover Approach:
- Convert all approved Purchase Requisitions to Purchase Orders
- Convert all approved Work Requests to Work Orders
- Close open transactions in the existing system
- Complete month end in the existing system
In practice, existing systems would be frozen up to two weeks prior to cutover with manual raising of items such as purchase requisitions and work requests during this period. The final Month End would run in the existing system and upon completion, the program would cutover to the new system, loading the data via the proven process.
If the cutover is successful, the system will be deemed live and business will resume operations on the new system by rekeying in any and all manual transactions, then Reconciling and Archiving the data, which will be covered in the next article in this series.
Takeaway #3
An integrated cutover plan is critical to the overall success of the program.
Data Standardising & Loading | Takeaways
- In all, six remedies need to be completed through this step of Data Migration. A reminder to commence Data Profiling and deleting Duplicate and Redundant data records as soon as possible so the focus of the Program can be on the quality and integrity of required data.
- The actual process of applying these remedies to Standardise the data for the new system is an iterative process consisting of multiple "Mock Runs"
- An integrated cutover plan is critical to the overall success of the program as it minimises the risks, the time, and the possible business disruption of cutover.
The next article in this series will focus on the next step - Data Reconciling & Archiving.
Download our Data Migration White Paper Series
Our four-part Business Guide to Data Migration covers the complete data migration lifecycle from pre-migration profiling through to post-migration archiving. Written by COSOL's Managing Director and CEO Scott McGowan, each paper draws on 25+ years of real-world delivery experience.
What the series covers:
- White Paper 1: Data Profiling and Remediation - how to assess your data and start remediation before a program begins
- White Paper 2: Data Standardisation and Loading - the six quality remedies and how mock runs work in practice
- White Paper 3: Data Reconciling and Archiving - how to handle legacy data and ensure production data integrity at go-live
- White Paper 4: Practical Application of Best Practices - real case studies from mining, energy, government, and transport organisations
About COSOL
COSOL is built on one belief: in asset-centric industries, reliability is everything. We're a trusted, data-led asset management partner for organisations around the world who can't afford to fail. And known for our deep expertise, dependable delivery, and ability to keep critical assets performing at their best.
The company recently celebrated 25 years in business and recognised as reliable partners by their clients across the globe.