Data engineering · Oil and gas
Production historian and real-time data integration
We build production historian and real-time data integration for oil and gas and process operations: read-only access to process historians and control systems, tag mapping to assets, time alignment, gap detection and filling, aggregation, and delivery to data platforms, production systems and analytics. Engineers get dependable time-series data in the form they need, and you own the pipelines and the code.
Requirements spec · Oil and gas
Upstream operations system requirements
A requirements specification you can adapt for an upstream production management or surveillance system: master data, acquisition, allocation, deferment, exceptions, reporting and the IT and OT boundary.- 01Eight requirement areas written as testable statements.
- 02Questions on data sources, quality flags and restatement rules.
- 03The IT and OT boundary items security teams ask about.
- 04Seven questions to put to any vendor or development partner.
Free download
Get it on screen in a few seconds.
What the system does, module by module.
- 01
Historian connectivity
Read access to process historians and OPC servers through their native interfaces, with connection management, authentication and backpressure. - 02
Tag inventory and mapping
A governed list of tags with description, unit, range and owner, mapped to assets, wells and facilities in the asset model. - 03
Streaming and batch extraction
Live subscription for current values and scheduled extraction of history, with backfill for new tags and for periods after an outage. - 04
Time alignment and resampling
Alignment of signals with different sampling rates and clocks, interpolation rules, time-zone handling and clear treatment of daylight saving. - 05
Data quality and gap handling
Detection of flatlines, spikes, out-of-range values and missing data, with quality flags, gap filling rules and an audit of what was changed. - 06
Aggregation and calculated tags
Averages, totals, min and max by interval, and derived measures such as rates and balances, calculated once and shared. - 07
Delivery to platforms and applications
Publication to data lakes, production systems, digital twins and analytics through APIs and streams, with consistent naming and units. - 08
Compression and storage strategy
Decisions on what to keep at full resolution and what to compress, with storage formats suited to analytics. - 09
Monitoring and operations
Health of each connection and pipeline, data latency and completeness, with alerts and runbooks.
Reference architecture.
Standards and interfaces
What we design against.
- OPC UA, HDA and historian APIs
- Standard and vendor interfaces for current and historical data, used read-only.
- IEC 62443
- Zones, conduits and secure development practice at the boundary between operational and business networks.
- ISA-95 hierarchy
- A model of enterprise, site, area and equipment for organising tags and assets.
- PRODML and OSDU time series
- Standards for exchanging production time series with partners and platforms.
- Columnar and time-series storage formats
- Parquet and time-series databases for efficient analytics on large histories.
- Time and unit conventions
- UTC storage with explicit local time handling, and consistent units with documented conversions.
Data model
The core entities.
- Tag
- Source, address, description, unit, range, sampling and owner.
- Asset mapping
- Tag to asset or well relationship, role and effective dates.
- Measurement
- Value, timestamp in UTC, quality flag and source.
- Aggregate
- Interval, statistic, value and completeness.
- Calculated tag
- Definition, inputs, version and the results produced.
- Gap and fill record
- Period, cause, fill method and the values changed.
- Pipeline run
- Source, window, records read and written, latency and errors.
When custom development is the right call.
Controls designed in, not added later.
Read-only by design
Connections use accounts that can only read, and no component can write to control systems.Network boundary
Extraction happens through a gateway or replica in a controlled zone, with firewall rules limited to what is needed.Integrity of history
Raw data is preserved. Corrections and fills are stored as separate, flagged data with a log.Secrets and certificates
Credentials and certificates are managed centrally and rotated.Monitoring
Alerts on stopped feeds, unusual volumes and failed authentication.
From discovery to a system you run.
- 01
Source and consumer survey
We list the historians, tags and sites, and the systems and teams that need the data, with operations, IT and automation engineers. - 02
Boundary and model design
The network boundary, tag model, naming, units and quality rules are agreed and approved by OT security. - 03
Connect one site end to end
One site is connected, mapped and delivered to a consumer, and the data is compared with operators' own views. - 04
Add quality, aggregation and consumers
Gap handling, calculated tags and further consumers are added. - 05
Roll out and operate
More sites follow with the same model, with monitoring and support in place.
Relevant work
Where to see our work.
Common questions.
- Can the integration ever write back to the control system?
- Not by design. Extraction uses read-only access, and the boundary does not allow inbound commands. Any exception would need explicit design, approval and agreement from the automation owner.
- How do you handle gaps and bad data?
- Raw data is kept as received. Quality checks flag problems, and any fill or correction is stored separately with its method and a log, so users can choose and auditors can trace.
- Does it work with more than one historian?
- Yes. Each historian is a connector behind the same tag model and delivery layer, so consumers see one consistent view.
- What about very high-frequency data?
- We decide with engineers what needs full resolution and what can be aggregated, and use storage formats suited to analytics so history stays affordable and fast to query.
- Who owns the pipelines and code?
- You do. The pipelines, the mappings and the code stay yours.