Organisations · Cralgo ·
Data quality problems rarely originate in the places where their consequences become visible. In most organisations, data enters through transactional operations—order entry, customer support ticketing, inventory movements, lead capture forms, or third-party integrations—and exits through reporting dashboards, analytical models, automated decisioning engines, or regulatory filings.
When discrepancies surface in executive dashboards or machine learning features, the initial response is typically structural: assign data quality accountability to the team experiencing the pain, usually a centralised data engineering or business intelligence team. When that fails to curb upstream errors, the mandate often shifts violently toward domain owners, insisting that software engineers or operational business units "own their data at the source."
Moving who is accountable for data quality changes the failure modes of the entire technical and operating system. Understanding when this accountability should shift, and what invariably breaks when it moves, requires inspecting the economic incentives, interface boundaries, and operational pressures governing each state.
The Centralised State: Downstream Compensation
In an early or mid-sized architecture, accountability almost universally rests downstream. Centralised data engineering teams build extract, transform, load (ETL) pipelines, ingest raw operational tables, and take full responsibility for serving clean data marts.
``` [Operational Apps / CRM] --> [Raw Ingestion] --> [Central Data Team] --> [Analytics / BI] (Emits messy data) (Schema changes) (Cleans, reconciles, (Consumes) takes accountability) ```
The Mechanism
The centralised model functions through downstream compensation. Upstream software engineers and operational teams optimize purely for feature delivery, system throughput, and transactional consistency. Because operational teams do not face operational penalties when an invalid status code enters a customer table, downstream data engineers deploy defensive pipelines. They introduce string normalization scripts, deduplication passes, default fallbacks, and reconciliation tables.
Why It Works Initially
This arrangement is rational when domain volume is low and downstream pipelines are simple. Centralising accountability in a single analytical unit provides:
- Global visibility into cross-system semantics (for instance, harmonizing how "churn" is derived across sales, finance, and product). - Specialised engineering capabilities in data modeling, dimensional warehousing, and SQL transformation. - Minimal friction for upstream application developers, who remain unencumbered by secondary analytical contracts.
How It Degrades
As an enterprise diversifies, the cognitive load on the central team outpaces capacity. When an operational engineering team refactors a relational database schema or alters an enum value to support a new user-facing checkout feature, downstream batch pipelines fail.
The downstream team lacks the domain context to evaluate whether an unexpected `null` in an order table indicates an abandoned transaction, a software defect, or a planned product experiment. The central team becomes an operational bottleneck, simultaneously blamed for broken operational dashboards and paralyzed by incomplete knowledge of upstream design intent.
The Push Upstream: Source-Aligned Accountability
Recognising that downstream teams cannot resolve upstream semantic drift, organisations frequently mandate a shift toward source-aligned accountability. Often framed around data mesh architectures or "data as a product," this model dictates that the teams producing the data are strictly accountable for its accuracy, schema integrity, and semantic validity.
The Trigger Points for Moving Accountability
Shifting accountability upstream becomes necessary only under specific structural conditions:
1. High Domain Divergence: When domain logic evolves faster than a central team can assimilate, downstream compensation becomes unmaintainable. 2. Asymmetrical Remediation Cost: When repairing a defective record in transit requires manual data stitching that costs significantly more than enforcing schema validation at the transactional write layer. 3. Operationalized Analytics: When downstream data feeds real-time decisioning systems, operational APIs, or machine learning inference engines where delayed batch reconciliation causes immediate financial or customer harm.
What Moving Accountability Breaks
Transferring accountability upstream is rarely an unalloyed improvement. Removing the downstream buffer exposes hard structural frictions between operational priorities and analytical requirements.
``` Upstream (Domain Teams) Downstream (Consumers) +-----------------------------------+ +-----------------------------------+ | Priorities: Uptime, throughput, | | Priorities: Trend continuity, | | sprint velocity. | | cross-domain joins, consistency. | | | | | | Burden: Publishing schema | | Risk: Semantic drift across | | contracts, backward compatibility | | uncoordinated producer boundaries| +-----------------------------------+ +-----------------------------------+ \ / \---- Interface Friction: -----------/ - Unfunded maintenance overhead - Semantic isolation between domains - Hidden operational drag ```
1. Operational Sprint Velocity and Prioritisation
Transactional application teams exist to ship user-facing capabilities, maintain system availability, and minimize latency on operational databases. Forcing software engineers to publish, document, and test stable analytical events introduces tangible drag.
When a sprint goal conflicts with maintaining backward compatibility for a data contract, local feature delivery almost always wins over remote data consumers. If source teams are given nominal accountability without structural adjustments to their team incentives and performance metrics, data contracts become shelfware.
2. Cross-Domain Semantic Consistency
Operational teams naturally view data through the lens of their bounded context. A billing service defines a "customer" by payment instruments and invoicing accounts; a support team defines a "customer" by contact history and identity tokens; an acquisition team defines a "customer" by an attribution cookie and lead state.
When data quality accountability moves entirely upstream, cross-domain semantic coherence collapses. Individual teams produce clean, locally valid datasets that cannot be joined downstream without complex, unowned transformation logic. Moving accountability upstream often solves syntactical data quality (types, nulls, ranges) while worsening semantic data quality (shared definitions and unified identity).
3. The Unfunded Infrastructure Burden
Demanding that operational units become accountable for data delivery requires them to maintain data infrastructure: event streams, schema registries, contract testing suites, and data publishing pipelines. Most operational software teams lack the specialized tooling and architectural background to run large-scale analytical distribution patterns reliably. The result is duplicated infrastructure, bespoke ad-hoc pipelines across domains, and developer attrition driven by unmanaged cognitive load.
Where Accountability Actually Belongs: The Contract Boundary
The failure of both total downstream absorption and total upstream devolution stems from treating data quality as a singular, indivisible responsibility.
Quality is an outcome of an agreement across a boundary. To Cralgo the governance of these systems is to acknowledge that accountability must be explicitly partitioned across the technical interface rather than handed whole to one side:
| Data Layer | Accountable Party | Specific Obligation | Failure Mode If Misassigned | | :--- | :--- | :--- | :--- | | Syntactic and Structural Integrity | Upstream Domain / Producing Team | Enforce schemas, prevent breaking contract changes, deliver within latency guarantees. | Downstream teams write brittle heuristics to patch malformed payloads. | | Semantic Interface Contracts | Shared Ownership (Producer + Consumer) | Explicitly define the business meaning of emitted events, status flags, and entity lifecycles. | Producer ships breaking business-logic changes disguised as valid types. | | Cross-Domain Harmonisation | Dedicated Core Platform / Integration Layer | Establish global identity graphs, entity resolution, and enterprise taxonomy. | Upstream teams get dragged into global corporate definitions they do not control. | | Fit-for-Purpose Usage | Downstream Consumer | Define analytical queries, business metric formulas, and appropriate aggregations. | Central team is blamed for business metrics derived from valid underlying data. |
Deciding When to Shift
Organisations should avoid moving accountability based on architectural fashion. An upstream shift is appropriate when the cost of communication failure between producer and consumer is lower than the cost of downstream data correction.
If operational teams are small and the core product is undergoing rapid structural iteration, enforcing rigid data contracts at the source will stifle product development. In this phase, maintaining a downstream engineering buffer that absorbs change is an intentional, acceptable tax.
Conversely, when an enterprise reaches the scale where downstream data teams spend over half their operational cycles triaging upstream regressions rather than building analytical capabilities, the buffer has failed. At that threshold, accountability must shift to explicit, testable contracts enforced at pull-request time in operational repositories.
The transition requires accepting the trade-off: analytical reliability increases, but transactional engineering cycles will absorb maintenance overhead. Moving accountability does not eliminate work; it simply dictates which side of the architectural boundary must bear the cost.
Explore the question of operational boundaries and technical systems design at Cralgo in our published work.
Frequently asked questions
What is the practical difference between syntactic and semantic data quality?
Syntactic data quality concerns the structure and format of information—such as column data types, non-null constraints, and valid ranges. Semantic data quality concerns whether the data accurately reflects the underlying business state and shares consistent definition across systems (for example, whether a 'cancelled subscription' means an immediate churn or a non-renewing active term). Upstream teams easily govern syntax, but shared semantics require cross-boundary coordination.
How does a data contract enforce accountability technically?
A data contract is a formal agreement between upstream service producers and downstream consumers that defines schema structure, SLA parameters, semantic definitions, and change-management protocols. It is enforced programmatically in software deployment pipelines (such as schema validation in continuous integration) rather than relying on informal documentation or retroactive communication.
When should an organisation keep data quality accountability centralised?
A centralised data team should continue to retain accountability during early product stages when operational schemas change weekly, when downstream consumer use cases are narrow, or when the operational cost of downstream pipeline repair is demonstrably lower than slowing core product development velocity.