Case Study: Evolving from Retrospective Analytics to AI-Ready Data Architecture...

This case study is designed for chief data officers, data architects, and data engineering leaders seeking to steer complex architectures through large-scale artificial intelligence transformations. It evaluates why moving beyond surface-level proof-of-concepts to establish unified semantic layers, policy-as-code governance, and product-centric data architectures is essential for eliminating technical friction and ensuring regulatory compliance. 

Introduction  

Modern organisations face a pivotal architectural crossroads driven by the exponential demand for artificial intelligence. This case study examines the shifts required to achieve enterprise-scale AI, derived from an expert briefing delivered to the DAMA Northern professional community by Savvi Vijayakumaraguru, Head of Data – Financial Services, and Tom Scott Coombes, Head of Data Engineering at Affinity Reply. As regulated institutions move beyond localised experimental applications toward deeply integrated decision intelligence, the responsibilities of data professionals have undergone a shift.  

The enthusiasm for Generative AI is undeniable. Industry benchmarks indicate that approximately 65% of organisations actively deploy GenAI capabilities within at least one business unit. However, beneath this rapid adoption lies a critical execution deficit. Gartner projections highlight that 30% of GenAI initiatives will be abandoned following initial proof-of-concept (POC) trials due to poor data quality, weak governance structures, and escalating implementation costs. Furthermore, through 2026, 60% of AI projects unsupported by dedicated, AI-ready data foundations are forecasted to fail. 

This dynamic illustrates the ‘Iceberg Paradox’ of modern transformation. Executive focus remains heavily fixated on the visible peak. So, the chatbots, copilots, interactive assistants, and stand-alone POCs. Yet, long-term value is dictated by the submerged, invisible structure of unified data foundations, operating models, automated governance, security controls, risk management frameworks, and direct process integration. Without modernising this underlying architecture, superficial AI initiatives likely won’t scale into dependable capabilities.

Legacy Structural Friction and System-of-Record Isolation

Legacy IT architecture across regulated industries is defined by a structural fragmentation that has been known to restrict agility. Information remains confined within isolated system-of-record silos, where core banking engines, CRM platforms, risk management databases, and actuarial systems independently own their transactional data. Extraction processes across these platforms are overwhelmingly manual, fragile, and ungoverned. As highlighted by Vijayakumaraguru and Coombes, this isolation stands as the primary obstacle cited across chief data officer surveys, resulting in a median request-to-delivery lead time of three to eight weeks for business-critical information.  

This isolation can foster widespread redundancy across technical teams. Lacking a shared catalogue or discoverable data assets, separate business units repeatedly construct identical metrics and key performance indicators. Within a typical Tier-1 banking institution, foundational entities such as ‘Customer 360’ models are routinely re-engineered between seven and twelve times. This duplicated effort consumes an estimated 40% to 60% of total analytical engineering capacity. Meanwhile, platform compute and storage costs grow without clear business attribution, with approximately 35% of cloud data expenditure absorbed by duplicated or entirely unused datasets.

Compounding this heavy financial load is the continuous pressure of complex regulatory mandates. Frameworks such as BCBS 239, GDPR, Consumer Duty, and DORA impose stringent, end-to-end data lineage requirements that legacy architectures cannot satisfy without extensive manual effort. Consequently, data teams remain locked in reactive compliance management, spending 70% to 80% of project timelines on basic data wrangling rather than high-value artificial intelligence development.

Creating ‘The Layer After Gold’

Once organisations realise that endless data wrangling and duplicated ‘Customer 360’ models are holding back their AI goals, the natural instinct is to turn to cloud data lakehouses. Engineering teams work hard to set up standard Bronze, Silver, and Gold Medallion patterns, assuming that curated, clean ‘Gold’ tables will solve their delivery bottlenecks. However, as Vijayakumaraguru and Coombes pointed out in their presentation, Gold tables only go halfway. While they do a fantastic job of structuring raw data, they remain static assets. Downstream users, whether a business analyst creating a report or an engineer building an ML feature table, are still forced to write custom logic, joins, and filters on top of those tables. Before long, business units are back where they started: creating slightly different calculations for the exact same metric.  

This is where the architecture evolves by introducing ‘the layer after Gold’, a unified semantic layer paired with a knowledge graph. 

Instead of letting every tool interpret raw tables in its own way, the semantic layer sits above physical storage as a shared translation engine. It defines core business logic, entities, and calculation rules just once. When a risk manager checks a certified executive dashboard, a self-service analyst explores data, or a generative AI agent queries the system, every one of them consumes the same business definitions without repeating complex technical pipelines.  

To give AI agents genuine understanding, modern platforms build a knowledge graph directly into this framework. If the semantic layer defines what a metric means, the knowledge graph maps how everything across the enterprise actually connects, linking customers, financial products, accounts, and regulatory rules in a searchable network. For large language models and retrieval-augmented generation (RAG) systems, this contextual map changes everything. By grounding AI models in real, structured business relationships, the graph eliminates hallucination risks, embeds security policies directly into data access, and turns static data stores into an active network for decision intelligence.

The Multi-Layer Data Product Stack

With a clear view of how semantic layers and knowledge graphs restore context to data, the next question is practical: how do we structure this into a repeatable, everyday operating model? Rather than building another huge pipeline, Vijayakumaraguru and Coombes recommend a modular reference architecture organised around the concept of managed data products. This layered blueprint creates a clear path from raw systems all the way to certified, AI-ready consumption.  

At the foundation sits Layer 01: Infrastructure, where cloud data platforms accommodate coexisting batch and streaming ingestion. Here, the familiar Medallion flow organises raw entries from Bronze to Gold. Crucially, to fix the cost allocation issues that plague legacy setups, compute and storage expenses are metered directly by data product, giving teams complete visibility into real ROI.  

Directly above sits Layer 02: The Product Layer, where data products are built, tested, versioned, and served. Transformation tools like dbt manage the build pipelines with clear lineage tracking, while data contracts are enforced automatically inside continuous integration (CI) environments. This safety net prevents breaking schema changes from ever reaching downstream business tools.  

To make these assets easily accessible across business units, Layer 03: The Data Marketplace provides an intuitive discovery portal. Rather than submitting manual tickets and waiting weeks for data extracts, business consumers can browse certified products, review usage history, and subscribe through automated entitlement workflows.  

At the top rests Layer 04: AI and Consumption, where end users unlock that sought-after business value. Autonomous AI agents query the semantic layer, risk managers consult certified dashboards, and quantitative models draw directly from ML-ready feature tables.

Tying all four layers together is a cross-cutting governance fabric operating as policy-as-code. Classification tags automatically drive role- and attribute-based access controls, enforce PII masking, maintain column-level security, and generate audit logs from metadata. By weaving data governance directly into the architecture, regulatory compliance ceases to be a manual task and becomes an automated byproduct of the day-to-day running of the organisation.

Technology Selection, Sovereignty, and Open Standards

Having mapped out this multi-layer data product stack, leaders may now ask themselves… how do we bring this architecture to life? As Vijayakumaraguru and Coombes emphasise, there is no single one-size-fits-all answer when evaluating build versus buy options. Every organisation carries a unique combination of legacy systems, internal capabilities, and priorities that must be weighed carefully.  

When selecting platforms and tools, engineering teams will likely prioritise seamless interoperability. Any new semantic tool or data catalogue ought to complement existing data warehouses and lakehouses, rather than becoming yet another isolated platform. In highly regulated sectors, this decision process is heavily guided by data governance. Architectural principles, strict jurisdictional guidelines, data sovereignty rules, and corporate risk postures often dictate where data can be processed and how tightly vendor ecosystems are controlled.  To protect these investments for the long haul, forward-thinking organisations actively anchor their platforms on open industry standards. By adopting emerging patterns such as the open semantic interface (OSI) for AI, enterprise teams ensure that their semantic models and business logic remain flexible and independent of any single vendor ecosystem. 

Key Takeaways for Data Professionals

It is inevitable that data teams will evolve their day-to-day practices to accommodate for AI modernisations. The insights shared by Savvi Vijayakumaraguru and Tom Scott Coombes offer a clear roadmap for technical leaders and data professionals looking to modernise their architectures:

Prioritise foundations over surface POCs: Generative AI tools and assistants cannot succeed at scale without trustworthy, governed, and contextualised data. Establishing semantic layers and knowledge graphs is essential for eliminating hallucination risk and giving autonomous agents real business context.  Eliminate redundant analytical effort: Stop re-engineering core entities and metrics like ‘Customer 360’ across different business units. Define calculation rules once within a centralised semantic layer and reuse them consistently across every dashboard, AI model, and regulatory report.  Treat data as a product: Structure technical delivery around modular, versioned, and contract-governed data products. Utilising a self-service marketplace gives business teams friction-free access to certified data assets while keeping storage and compute costs completely transparent.  Embed automated governance: Replace manual access approvals and spreadsheet-based lineage tracking with policy-as-code. Enforcing automated classification, PII masking, and audit logging directly inside the engineering pipeline turns regulatory compliance into a seamless byproduct of normal operations.   

--------------------------------------

Ultimately, modernising for artificial intelligence is not necessarily about acquiring newer software. It is about changing how organisations value, structure, and govern their core information.

This article is linked to the latest Northern Community Meet Up in June 26. click here for the report.