Data engineering · Migration
ERP data migration and data quality
We carry out ERP data migration and data quality work for enterprises moving to a new ERP or consolidating several: profiling and cleansing of master and transactional data, mapping and transformation to the target model, repeated mock loads, reconciliation of balances and open items, cut-over rehearsals and archiving of what stays behind. Migration becomes a repeatable, measured process, and you own the migration code and rules.
Reference architecture · Data and integration
Data platform and integration reference architecture
A reference checklist for enterprise integration, modernisation and data platform projects: domains, integration patterns, contracts, platform layers, master data, quality, security and operating model.- 01Eight areas to decide before you modernise, integrate or build a data platform.
- 02Contract, schema and versioning items that stop integrations breaking.
- 03Quality and ownership items that make a platform trusted.
- 04Seven questions to put to any vendor or development partner.
Free download
Get it on screen in a few seconds.
What the system does, module by module.
- 01
Scope and data inventory
A list of objects to migrate, from customers and materials to open orders and balances, with volumes, owners, quality and what can be archived instead. - 02
Data profiling
Analysis of each source for completeness, duplicates, invalid values, inconsistent codes and rules that the data breaks, with quantified findings. - 03
Cleansing and harmonisation
Correction, de-duplication and standardisation of master data, with business owners deciding on conflicts and a record of every change. - 04
Mapping and transformation
Mapping of source fields, codes and structures to the target, written as testable rules, including defaults, derivations and value conversions. - 05
Load automation
Automated extraction, transformation and loading using the target ERP's migration tools and APIs, rerunnable and with error handling. - 06
Mock loads and testing
Repeated full-volume trial migrations with business testing of the loaded data, tracking of defects and improvement of each cycle. - 07
Reconciliation
Automated comparison of counts, balances, stock and open items between source and target, with differences explained and signed off. - 08
Cut-over planning and rehearsal
Detailed cut-over plans with timings, dependencies and rollback, rehearsed so the final weekend holds no surprises. - 09
Archiving and decommissioning
Archive of historical data that is not migrated, in a searchable form, and safe retirement of source systems.
Reference architecture.
Standards and interfaces
What we design against.
- ISO 8000 and ISO/IEC 25012
- Data quality standards, including the quality of master data, that define the dimensions measured.
- DAMA-DMBOK
- Data management body of knowledge, used for roles, governance and data quality practice.
- Target ERP migration tools
- The vendor's migration objects, templates and interfaces, with extensions where they fall short.
- Accounting and statutory requirements
- Rules on balances, open items, retention and audit access that the migration must respect.
- Data protection law
- Treatment of personal data in migration, test environments and archives.
- Test and release practice
- Defect management, entry and exit criteria and sign-off for each test cycle.
Data model
The core entities.
- Migration object
- Object type, source, target, volume, owner, rules and status.
- Profiling finding
- Rule broken, field, count, examples and decision.
- Mapping rule
- Source field or code, target, transformation, version and tests.
- Cleansing decision
- Record or group, action, approver and rationale.
- Load run
- Cycle, object, records read, loaded and rejected, and errors.
- Reconciliation result
- Check, source value, target value, difference, explanation and sign-off.
- Cut-over step
- Task, owner, duration, dependencies, outcome and rollback condition.
When custom development is the right call.
Controls designed in, not added later.
Protected test data
Personal and financial data is masked in non-production environments, with controlled access to full copies.Controlled access to staging
The staging area is restricted, logged and cleared after cut-over.Approval of cleansing
Business owners approve merges and deletions of master data, with the decision recorded.Reconciliation evidence
Signed reconciliation results are kept as evidence for finance and auditors.Rollback readiness
Rollback conditions and steps are defined, tested and agreed before cut-over.
From discovery to a system you run.
- 01
Inventory and profiling
We inventory the data to migrate and profile it, so the programme knows the true state of quality before design is fixed. - 02
Rules and tooling
Mapping rules, cleansing approach, tools and reconciliation checks are designed with the business and the ERP team. - 03
First mock load
A full-volume trial migration is run, tested by the business and the results used to improve rules and data. - 04
Further mock loads
Mock loads repeat until quality and timing meet the agreed standard, with a report for each cycle. - 05
Cut-over and archive
Rehearsed cut-over, final reconciliation and sign-off, followed by archiving and decommissioning of the old systems.
Relevant work
Where to see our work.
Common questions.
- Why do ERP migrations run late?
- Often because data quality is discovered late. Profiling before design, repeated full-volume mock loads and business testing of loaded data find the problems early, when they are cheap to fix.
- How many mock loads do we need?
- As many as it takes to reach the agreed quality and timing. Each one should be better than the last, and the reports show whether that is happening.
- Who decides which duplicates to merge?
- Business owners do. We propose matches and rules, and owners approve merges and deletions, so the data reflects business decisions with a record of each one.
- What happens to old data that is not migrated?
- It is archived in a searchable form with the same access controls and retention, so auditors and users can still find it after the old system is retired.
- Who owns the migration code and rules?
- You do. The code, the mappings and the rules stay yours.