Article by COSOL (Data Team)
Our Point of View
Experience shows that one critical success factor for these programs to realise the value in their business cases is Data Migration. The axiom "garbage in, garbage out" remains as relevant today as it did when computers were first invented. COSOL strongly believes:
that digital transformation is well underway, and every Board is, and should be worried about how to become a truly digital enterprise.
strong Enterprise Data foundations will be required to enable adoption of digital solutions including advanced analytics, robotic process automation, machine learning and artificial intelligence which are the next frontiers to productivity and market competitiveness.
Enterprise Data is the glue, the fact base, that drives decision making and business improvement, allowing organisations to meet stakeholder expectations in a timely and efficient manner; and
for organisations to succeed, Data must be treated as a mission critical Asset; it is the single biggest success factor in a digital transformation journey, and most organisations are ill prepared due to many islands of disconnected data that is of unknown and/or poor quality.
Introduction to Data Migration
Sadly, data only gets into the spotlight when an organisation Is faced with a large system upgrade. Unfortunately, this is typically in a negative light when it is exposed as a major risk to a program with a major cost attached to remediate the data and mitigate the risk. This is a symptom of being under-valued in a traditional business model that has survived on paper-based systems and manual data entry which is not sustainable in the digital age.
As a “domain”, data should and can be managed before, during and after a major program.
This strengthens organisations overall digital capability and mitigates future risks and costs by ensuring data remains evergreen.
Data Migration can be best explained as three distinct steps:
| Pre-Migration | During Migration | Post-Migration |
|---|---|---|
| Step 1 Data Profiling & Remediation |
Step 2 Data Standardisation & Loading |
Step 3 Data Reconciling & Archiving |
This article focuses on the Pre-Migration step.
The Pre-Migration step is preparing your organisations data for migration. Establishing Data Owners who can then contribute to and approve Data Quality Management (DQM) objectives and guiding principles will define the profiling and remediation requirements of the pending data migration.
Data Profiling is the systematic analysis of the source data based on the requirements given. The desired outcome of the Profiling activity is to provide a correct and complete model for the target solution. Once data has been assessed, the results of the Profiling will drive the Data Remediation by determining the actions needed to cleanse and prepare the source data.
Data Remediation can involve correcting duplicate, incomplete, inaccurate, and corrupt records in existing systems. There may also be a requirement for the standardisation and harmonisation of records to align with the requirements of the new system. There are six dimensions of data quality as outlined in Table 1: Six Dimensions of Data Quality shown in the chapter In-Program vs. Pre-Program Data Remediation.
Takeaway #1
Data will typically be exposed as a major risk and cost in large scale digital transformations due to unknown and/or poor quality.
When & Where to Start?
A common practice and misconception in traditional, less digitally mature business is that a program, together with some technology, will fix whatever issues are encountered, and once fixed It will be fixed for good.
- The program risks and costs increase significantly due to a lack of data Readiness; and
- The program will, by nature, only focus on doing the minimum work required for the program to achieve its objectives, often described as a new system going live. This approach most often does not include ensuring the business has a Sustaining capability and as a result, data will sadly fall into a state of disrepair over time, and the benefits of the transformation will not be sustained.
Takeaway #2
Strong Data Ownership and Data Governance is Critical
A better/best practice is to acknowledge that this Is a business issue first and foremost and that the business must take ownership and accountability of its data as a strategic asset.
For each Master Data object (Finance, Supplier, People etc), a senior business person, typically from the relevant business function, should be assigned the role and responsibilities of a Data Owner (see Appendix A for a guide on roles & responsibilities).
Data Owners, along with the organisations Process Owners should meet regularly in an Enterprise Process and Data Owners forum to discuss, review and agree on performance, quality and remediation activities needed to continually improve business performance.
In-Program vs. Pre-Program | Data Remediation
As previously mentioned, a common misconception is that program commencement is needed to kickoff Data Profiling & Remediating.
Per Figure 3: Common vs. Best Practice Approach to Data Migration, a better practice is for Data Owners to commence Data Profiling as soon as practical to understand the baseline quality of their data.
Data Quality can be improved ahead of a Data Migration program which reduces time, effort, cost and risk to the actual program, and further strengthens the sustaining capability to maintain data quality beyond the program completion.
Initial Data Quality efforts should focus on the top 3 Dimensions of Data Quality Dimensions shown here in Table 1: Six Dimensions of Data Quality.
Takeaway #3
Commence Data Profiling & Data Remediating as early as possible.
| Business | Duplication | Duplicate records may exist (e.g. specific master data may exist multiple times with each instance having a variation to the original master data naming convention). Any duplicates are to be deleted and relationships amended to link the surviving row. |
| Redundancy | Data that is no longer current. These should be identified in source systems and corrected accordingly. (e.g., Active vendors without an invoice in the last 14 months). | |
| Program | Standardisation | The data to be migrated will need to conform to an approved or conventional standard (e.g., Master Data Standard). Cross-validation of datasets across table structures against the agreed standards will need to be monitored and rectified as appropriate until go-live. |
| Incorrect | The data has the incorrect business value (e.g., Entitlement amount is incorrect, Bank Account name is incorrect, Addresses inclusive of post codes are incorrect, field values are not aligned to their original planned usage patterns). | |
| Integrity | Relationships are not maintained correctly, such as orphaned records (e.g., an Account Balance for an Account that no longer exists). Tables and application functional areas must be maintained (e.g., Organisational Structure, Accounts, Reports to etc.). | |
| Completeness | Is a measure of data content quality expresses as a percentage of the columns or fields of a table or file that should have values in them, or fields that need to be left blank and have values in them (e.g., First Names not in Preferred Name column). |
Data Profiling & Data Remediating | Takeaways
- Data will typically be exposed as a major risk and cost in large scale digital transformations due to unknown and/or poor quality. If you don't know the quality of your data today, this will likely be true for you.
- Improve your Readiness by:
- establishing strong Data Ownership and Data Governance immediately. These roles are needed now and will be required into your digital future. Start now and begin to develop that capability.
- ensuring the Data Owners define Data Quality objectives as their priority task. These objectives are used for profiling the data and baselining your current situation. Without this, your organisation and any pending programs are flying blind.
- starting Data Remediation as soon as possible. This is the most common pitfall, where organisations leave It to the 'program' to fix the problem. Data Owners can, and should, resolve duplicate data and redundant data in existing systems. There is no dependency on new systems for this to occur, and this exercise alone helps develop a critical enduring capability.
We hope that these guidelines help you in some way with your digital transformation journey in this critical, yet often overlooked area of Data Migration.
The next article in this series will focus on the next step - Data Loading.
Appendix A | Data Owners and Data Steward Roles & Responsibilities Guide
| Data Owners | Data Stewards |
|---|---|
|
Business Role
|
Business Role
|
|
Key Responsibilities
|
Key Responsibilities
|
Download our Data Migration White Paper Series
Our four-part Business Guide to Data Migration covers the complete data migration lifecycle from pre-migration profiling through to post-migration archiving. Written by COSOL's Managing Director and CEO Scott McGowan, each paper draws on 25+ years of real-world delivery experience.
What the series covers:
- White Paper 1: Data Profiling and Remediation - how to assess your data and start remediation before a program begins
- White Paper 2: Data Standardisation and Loading - the six quality remedies and how mock runs work in practice
- White Paper 3: Data Reconciling and Archiving - how to handle legacy data and ensure production data integrity at go-live
- White Paper 4: Practical Application of Best Practices - real case studies from mining, energy, government, and transport organisations
About COSOL
COSOL is built on one belief: in asset-centric industries, reliability is everything. We're a trusted, data-led asset management partner for organisations around the world who can't afford to fail. And known for our deep expertise, dependable delivery, and ability to keep critical assets performing at their best.
The company recently celebrated 25 years in business and recognised as reliable partners by their clients across the globe.