Data Engineering and Data Platforms: the layer every model quietly depends on

When a production AI system produces a confidently wrong answer, the instinct is to blame the model. In practice, the more common root cause sits two layers below: stale data, an untracked schema change, a missing access-control mapping, or a transformation error that silently corrupted a field the model or retrieval system depends on. Data engineering at AI scale is not the traditional data-warehouse discipline with a new label; it has to support both batch and streaming freshness requirements, feed both classical analytics and embeddings-based retrieval, and carry lineage and quality metadata all the way to the point where a model or agent consumes it.

Enterprise data estates are typically fragmented across operational systems, legacy warehouses and newer lakehouse platforms, with inconsistent quality controls and incomplete lineage. AI initiatives expose this fragmentation faster and more visibly than traditional BI ever did, because a retrieval system or agent that reasons over bad data produces an answer that looks authoritative and is wrong, rather than a report that is simply late.

This layer sits above Layer 02/03 (which it may run on and call, respectively, for embedding generation or transformation-time inference) and directly beneath Layer 05 (Vector Search), which depends entirely on this layer's output quality, freshness and access-control metadata. It is also a direct input to Layer 08 (LLM Engineering), since fine-tuning and evaluation both require curated, well-governed datasets.

Batch and streaming ingestion; transformation and enrichment; data quality validation and monitoring; cataloguing and metadata management; lineage tracking; orchestration and scheduling; storage architecture (lakehouse patterns unifying analytics and AI workloads).

Apache Spark remains the dominant engine for large-scale batch and increasingly streaming transformation, chosen for its maturity, ecosystem and ability to run across cloud and on-premises infrastructure. Databricks packages Spark with a managed lakehouse platform, notebook-based development and governance tooling, favoring speed of delivery for teams that accept a more managed, less infrastructure-flexible model; its own documentation describes the "medallion" bronze/silver/gold pattern as a way to progressively improve data quality, which is a useful organizing convention but not, on its own, a governance program.¹ Apache Airflow is the leading choice for batch pipeline orchestration where complex dependency graphs and scheduling need to be explicit and auditable. Apache Kafka is the standard for high-throughput streaming and event-driven data movement, essential wherever an AI system needs near-real-time freshness (fraud detection, operational monitoring) rather than daily batch refresh. dbt (data build tool) has become the standard for SQL-based, version-controlled, testable transformation logic inside the warehouse or lakehouse, valued specifically for making transformation logic auditable and testable, with lineage and cataloguing surfaced directly from the transformation project rather than maintained separately.²

Managed (Databricks) versus self-operated (Spark on CoralDC-provisioned infrastructure) trades delivery speed against infrastructure control and, for sovereignty-constrained workloads, against the ability to guarantee data never leaves a specific jurisdiction. Batch versus streaming architecture is a freshness-versus-complexity trade-off: streaming pipelines are materially more complex to build and operate correctly, and should be reserved for workloads where the latency actually matters to the business outcome. Centralized data platform versus federated/domain-owned data products (a data-mesh-style pattern) trades consistency and governance simplicity against organizational scalability in very large, multi-business-unit enterprises.

Row- and column-level access control that survives into downstream consumption (including the vector index at Layer 05); automated data quality testing as a pipeline gate, not a manual review; full lineage tracking from source system to model input, sufficient to answer "which raw data produced this model's training set or this retrieval result"; catalogued, discoverable datasets with owner and sensitivity classification; and encryption in transit and at rest matched to the data's classification and the deployment's sovereignty requirement.

Discover (data estate inventory, quality baseline, freshness requirements by use case); architect (pipeline and storage design, lineage and quality tooling selection); prototype (a representative pipeline validated end to end into a downstream AI use case); industrialize (orchestration, testing, monitoring and access control wired in); deploy; govern (ongoing data quality and access review); operate (pipeline reliability, incident response); optimize (cost and performance tuning as data volume and use-case count grow).

Financial services requires strict lineage and access control for regulatory reporting and model-risk purposes, frequently under DORA or equivalent operational-resilience obligations in the EU/UK. Healthcare and life sciences require de-identification and consent-aware pipelines feeding both analytics and AI use cases. Manufacturing and energy operators increasingly need streaming pipelines from IoT and sensor data to feed predictive-maintenance and digital-twin applications at Layer 10.

Data quality pass rate at pipeline gates; time from source-system change to available, validated downstream data; percentage of production AI use cases with complete lineage traceability; pipeline reliability (failure and recovery rate).

Building a retrieval or fine-tuning use case directly against raw, ungoverned data because it is faster to start; treating data quality as a one-time cleanup rather than a continuous, tested pipeline gate; allowing access control to be enforced only at the source system and assuming it survives, unenforced, into every downstream copy and index; and defaulting to a streaming architecture for use cases that do not actually need sub-hour freshness, adding operational complexity with no corresponding business benefit.

Scignal's data engineers design pipelines that carry governance metadata, quality gates and lineage all the way into the retrieval and model layers above, running on CoralDC infrastructure chosen to match the data's actual sovereignty and residency requirement rather than a default cloud region.