Data engineering · Banking
Banking data warehouse and lakehouse
We build banking data warehouses and lakehouses: governed ingestion from core banking, payments, cards, CRM and risk systems, domain models for customers, accounts and transactions, lineage and quality controls strong enough for regulatory reporting, and serving layers for analytics, machine learning and real-time use. The bank has one trusted source for its data, and you own the platform and the code.
Requirements pack · Banking finance
Bank reporting and controlling requirements pack
A requirements pack for regulatory reporting, treasury and liquidity reporting, and management controlling systems in a bank: data, definitions, calculations, validation, lineage, close and governance.- 01Eight requirement areas shared by regulatory, treasury and controlling reporting.
- 02What to ask for on data definitions, lineage and reconciliation.
- 03Close, sign-off and change-management items auditors check.
- 04Seven questions to put to any vendor or development partner.
Free download
Get it on screen in a few seconds.
What the system does, module by module.
- 01
Source ingestion
Change data capture, files and APIs from core, payments, cards, loans, CRM and market systems, with replay and a record of exactly what was loaded and when. - 02
Raw and curated layers
An immutable raw layer that mirrors sources, and curated layers where data is conformed, historised and documented, so each question has one answer. - 03
Banking domain model
Customers, parties, accounts, products, transactions, collateral and exposures modelled against industry reference models, with history and effective dates. - 04
Data quality controls
Completeness, validity, consistency and reconciliation checks at ingestion and in curated layers, with quality scores, owners and escalation. - 05
Lineage and catalogue
A searchable catalogue with business definitions and technical lineage from source field to report, available to analysts, risk and auditors. - 06
Security, masking and entitlements
Classification of sensitive data, column and row-level controls, masking and tokenisation, and access requests with approval and expiry. - 07
Semantic layer and data products
Shared metric definitions and governed data products for finance, risk, marketing and operations, so the same figure is the same everywhere. - 08
Streaming and real-time features
Event streams for fraud, limits and customer interactions, with features computed once and used in both batch and real-time models. - 09
Machine learning and analytics enablement
Feature stores, notebooks and model training on governed data, with reproducible datasets for model validation.
Reference architecture.
Standards and interfaces
What we design against.
- BIAN service landscape
- The Banking Industry Architecture Network's service domains, used as a reference for organising the data model and the ownership of domains.
- FIBO
- The Financial Industry Business Ontology, a shared vocabulary for financial concepts that helps align definitions across systems.
- ISO 20022 business model
- The payment and securities data dictionary, used to keep rich payment data intact in the platform.
- BCBS 239
- Principles for risk data aggregation and reporting that govern data quality, lineage and timeliness for risk data.
- Open table formats
- Delta Lake, Apache Iceberg or Apache Hudi for transactional tables, schema evolution and time travel on lake storage.
- Data protection and banking secrecy law
- Controls on personal and customer data, cross-border access and retention.
Data model
The core entities.
- Party and customer
- Identity, roles, relationships, segmentation and history of each version.
- Account and product
- Account attributes, product terms, status and the party relationships.
- Transaction and payment
- Posted entries and payment messages with their parties, amounts, purposes and references.
- Exposure and collateral
- Limits, balances, guarantees, collateral and the links between them.
- Data product
- Definition, owner, schema, quality score, consumers and service level.
- Quality rule and result
- Rule, scope, threshold, outcome, owner and remediation.
- Lineage edge
- Source and target element, transformation and run identifier.
When custom development is the right call.
Controls designed in, not added later.
Classification and masking
Sensitive data is classified at ingestion, and masking, tokenisation and encryption apply by class.Fine-grained access
Row and column-level policies by role and purpose, with access requests approved and expiring.Residency and sovereignty
Data held in the regions that law and policy require, with controls on cross-border access.Auditability
Every access, change and pipeline run is logged and reproducible.Resilience
Tested recovery and replay of ingestion, with immutable raw data as the backstop.
From discovery to a system you run.
- 01
Data landscape and priorities
We inventory sources, consumers and pain points with finance, risk, regulatory reporting and analytics, and rank domains by value and regulatory need. - 02
Architecture and governance design
Layers, models, ownership, quality rules and security are agreed, together with the platform and the operating model. - 03
Build the first domain end to end
One domain, usually customers and accounts, is built from ingestion to a governed product, with lineage and quality from the start. - 04
Add domains and consumers
Transactions, exposures and others follow, and regulatory, risk and finance consumers move onto the platform. - 05
Operate and evolve
Platform operations, cost control and a product-based roadmap, with new domains and real-time features added.
Relevant work
Where to see our work.
Common questions.
- Warehouse or lakehouse?
- A lakehouse combines warehouse-style tables and governance with cheap storage for raw and unstructured data, and is the usual choice today. What matters more than the label is the governance, the domain model and the quality controls.
- Can the platform serve real-time use cases?
- Yes. Event streams feed real-time features and decisions, with the same definitions used in batch, so fraud and limits logic do not drift from reporting.
- How do you keep data quality high?
- Quality rules run at ingestion and in curated layers with owners, thresholds and escalation, and reconciliations to the ledger prove that the numbers agree with source systems.
- Does it replace our regulatory reporting system?
- No. It supplies governed data to it. Many banks use the platform as the single data source for regulatory, risk and finance reporting.
- Who owns the platform and code?
- You do. The code, the models and the data stay yours.