Data architecture · Lakehouse · Data governance
The platform is easy. The architecture is the work.
Databricks, Snowflake and the cloud data platforms have made it trivially easy to land data and build on it. That's exactly the problem. Without a data architecture — and someone accountable for it — the lakehouse becomes a swamp in about a year, and the AI programme stalls on top of it. We design the architecture first — as chief architects who have built data platforms on Databricks, Snowflake, Azure and GCP and finished five migrations — with security and governance as properties of the structure rather than things to retrofit.
The tell is always the same.
Ask three people in the business what the source of truth for revenue is. If you get three answers, you don't have a data architecture — you have a data platform with opinions.
What happens without it
Six things that go wrong — and what the architecture fixes.
These arrive in roughly this order. Most organisations recognise the first two; by the time the fifth bites, the AI programme is already a year late.
Without a data architecture
- The swamp. Everyone lands everything. Six copies of "customer" with six definitions; bronze is a dumping ground; medallion layers are folder names, not contracts.
- Notebook sprawl. Exploration becomes production. No lineage, no versioning, and a critical report depends on a notebook written by someone who's left.
- Cost runs away. Clusters left running, nothing incremental, transforms recomputed from raw. The bill becomes a board surprise and the platform gets the blame.
- Security retrofitted, badly. Access granted ad hoc at table level; PII unclassified in silver and gold; nobody can say where personal data flows when the DPIA arrives.
- The AI programme stalls. Agents and RAG need quality contracts, lineage, a glossary and a trusted gold layer. "Where did this training data come from?" gets a shrug.
- The migration never finishes — or migrates too much. Nothing was modelled, so no one can say what's moved, and ex-customers get copied along with everyone else. Two platforms run in parallel indefinitely, reconciling between them.
What a data architecture provides
- One model of the business. A target data architecture and domain model; a glossary so "customer" and "revenue" mean one thing everywhere.
- Pipelines, not notebooks. Medallion contracts that mean something; versioned, incremental, lineage-tracked data products with named owners.
- Cost by design. Workload tiering, cluster policies, incremental patterns and FinOps guardrails set before the first bill, not after the tenth.
- Governance as structure. Unity Catalog designed up front — catalogs by domain, schemas by product, access by group; classification, row and column security, retention and DLP built in.
- AI-ready by default. Quality contracts, lineage and a trusted gold layer: the prerequisites AI governance will ask for, answered before it asks.
- A migration that ends. Modelled scope, a known inventory, a decommissioning plan — one platform, one bill, one truth.
What a data architect produces
Six artefacts. If you don't have them, you don't have a data architecture.
A data architecture is a set of models, policies, rules and standards that maps the data estate and governs which data is created, where it lives and how it flows. In practice it is six documents, each with a defined audience. They are built in order; each one feeds the next.
Conceptual data model
The highest level of the hierarchy: related entities grouped by subject area — Customer, Product, Order, Site. One page. It is the drawing the business uses to understand its own data landscape and to agree who owns what.
Logical data model
Every entity, its attributes, and how they relate — logically, independent of any platform. This is what the process and solution teams design against, so that two teams building on the same lakehouse don't invent two Customers.
Data dictionary & glossary
The entities and attributes with definitions, characteristics, business rules and standards — and, in our version, classification and retention per term. Used by the business to understand its own terms and by builders to get them right.
CRUD matrices
Entities and attributes plotted against the processes and applications that create, read, update and delete them. Produced from the dictionary and the models. This is where "which system is the source of truth?" stops being an opinion.
Physical data model
The detailed, platform-specific model: catalogs, schemas, tables, keys, partitions, the medallion layers. The only artefact most Databricks estates have — built without the four above it, which is why it drifts.
Principles & standards
The enduring guidelines and values that govern design, management and decision-making: naming, keys, classification, retention, access models, what "done" means. Short, numbered, and quoted in every design review.
What we design
Four layers, designed together.
Most data architecture is done one layer at a time by different people. We design all four as one system, using the SABSA method below — which is why security and governance end up built in rather than bolted on.
Domain & data model
Business domains, conceptual and logical data models, the glossary, and the data products each domain owns. The map everything else is built on.
Lakehouse architecture
Medallion design, ingestion and integration patterns, warehouse and hub design, workload tiering and FinOps — on Databricks, Snowflake, Azure or GCP.
Catalog, classification & access
Unity Catalog or equivalent structure, data classification, access model, lineage, retention and DLP. Governance you can audit because it's in the structure.
Ownership & operating model
Data product owners, stewardship, a design authority, and the standards that let the platform team say no — so the architecture survives the people who built it.
Accelerators
We don't start a lakehouse from a blank workspace.
A set of accelerators, built across previous platforms and refined on each one, that fast-track a lakehouse to right-sized, secure and compliant — so the first weeks go on your domain model, not on re-deciding how a catalog should be laid out.
Landing zone & catalog blueprint
Pre-built Unity Catalog structure (catalog by domain, schema by product), workspace layout, medallion templates and pipeline patterns, ready to take a domain model.
Workload tiering & FinOps guardrails
Cluster policies, serverless and Photon decision rules, incremental patterns and cost reporting from the first run — so the bill is designed, not discovered.
Security pattern library for data
The four-pattern access model below, row and column security, masking and tokenisation patterns, service identities for pipelines, key management and private-link designs, audit to SIEM.
Classification, retention & deletion kit
Classification scheme mapped to GDPR, HIPAA, SS1/23 and DORA; retention schedule templates; tenant-keyed deletion designed in, including embeddings and backups; DPIA-ready lineage.
Access models
Who can see what — decided at the logical layer, enforced at the physical.
Most lakehouses grant access table by table, on request, until nobody can explain who sees what. We design the access model once, from the domain and classification model, and let the platform enforce it. Four patterns, layered rather than chosen between:
Role & group grants
Access by group, never by individual: groups mapped to roles, roles to catalogs and schemas. Joiners, movers and leavers are an identity change, not a data-platform ticket.
Row & column security (ABAC)
Policies on data attributes — tenant, region, classification, sensitivity — applied automatically as filters and masks. One policy covers every table carrying the attribute.
Tenant-keyed segregation
Tenant ID on every row and derived object; isolation enforced by policy, not by trust; the same key that drives segregation drives deletion. Multi-tenant by design, single-tenant where regulation demands it.
Pipelines, services & AI agents
Every pipeline, service and agent gets its own identity with least privilege and scoped, short-lived credentials — never a human's. What a model may read is a policy decision, recorded in its model card.
Method · SABSA
Business-driven, from the top layer down.
We design data architecture the way we design security architecture — using SABSA, so the two end up on one drawing. Three principles do most of the work:
Security on the same drawing
Your data model is your information asset catalogue. We only draw it once.
Security architecture needs a catalogue of information assets, valued for confidentiality, integrity and availability. Data architecture produces exactly that list — the conceptual and logical models. Done separately, you get two catalogues that disagree. Done together, every domain carries its classification from day one and every control can be traced to a reason.
One catalogue, two uses
If an organisation already has a data model, it is the basis of the information asset catalogue. We don't build a second list for security; we add three columns to the first one.
Group, don't enumerate
An enterprise has millions of data items. We classify at domain and entity level, and only go to attribute level where the classification actually changes — PII, payment, secrets.
Classification drives the policy
C-I-A on the glossary determines the access model, the retention, the encryption and the audit depth. The platform enforces it; it doesn't decide it.
The stakes
Reputational loss, IP loss, lost customer confidence, interruption costs, and regulatory fines of up to 4% of global turnover. The catalogue is how you know which of those a given table could cause.
Where are you on the ladder?
Five levels. Most lakehouse estates are at one or two, and think they're at four.
We assess five components in the first two weeks of any review. The platform usually scores well. The governance, quality and methods around it usually don't — and that is what decides whether the AI programme on top of it is safe.
Manual, push information model on the left · automated, self-service, pull model on the right.
Why the foundation decides the AI
You cannot build the top of this without the bottom. Most people try.
Each tier only delivers if the one below is trustworthy. An AI programme sitting on an unconsolidated, ungoverned foundation doesn't fail loudly — it produces confident, wrong answers, at scale.
Data reliability
A comprehensive data management capability — governance, quality, security, architecture, lifecycle, MDM — so the business can have confidence in its own landscape. This is the tier that gets skipped.
Accurate information
Data from varied, complex sources brought together for query: the lakehouse or warehouse, built to the logical model, one key per domain, retention and classification carried through every layer.
Proactive insight
Analytics as a prime source of decision support — predictive and prescriptive, self-service, with lineage back to a system of record so the number can be defended.
Trusted decisions
Models and agents consuming only classified, owned, retention-tagged data; access via the same groups and policies as everyone else; logged for as long as the regulator says. See AI governance.
What it looks like in practice
One domain, traced end to end.
Traceability is the principle; this is what it produces. One business attribute for the Revenue domain, followed from what the business asked for to the platform object that delivers it — with ownership, classification and retention decided on the way, not afterwards. A pattern, not a client's data.
The glossary is where "one definition" becomes real. Ten terms from a typical build — note the columns most glossaries don't have.
Swipe the table sideways on a phone.
| Term | Definition | Domain · owner | System of record | Class | Retention |
|---|---|---|---|---|---|
| Customer | A legal entity or person with at least one active or historic contract | Customer · Head of CRM | CRM | Internal | Contract + 7y |
| Account | A billing relationship under a customer; one customer may have many | Finance · Billing Lead | Billing platform | Internal | Contract + 7y |
| Recognised revenue | Revenue recognised under IFRS 15 in the period, net of credits | Revenue · Finance Data Lead | General ledger | Confidential | 7y |
| Booking | Contracted value at signature; not revenue until recognised | Sales · RevOps Lead | CRM | Confidential | 7y |
| Active user | A named user with an authenticated session in the trailing 30 days | Product · Product Analytics | Identity provider + product events | Personal | 24 months |
| Contact | A person associated with a customer, with a consented channel | Customer · Head of CRM | CRM | Personal | Consent + 24 months |
| Supplier | A legal entity we pay under contract or purchase order | Procurement · Procurement Lead | ERP | Internal | Contract + 7y |
| Employee | A person on payroll or an active contractor record | People · HR Systems Lead | HRIS | Personal · special category possible | Employment + 6y |
| Incident | A recorded security or operational event with a severity and an owner | Security · CISO | ITSM / SIEM | Confidential | 6y |
| Model (AI) | A versioned AI model or agent with a model card, owner and risk tier | AI governance · Model Owner | AI registry | Confidential | 10y (EU AI Act Art 18) |
Quality, lineage, a trusted gold layer
Agents and RAG inherit whatever the data has. Without quality contracts, lineage and classification, "where did this training data come from?" gets a shrug — and the AI governance framework can't sign it off.
The prerequisites, already answered
The glossary says what the data means; the data product says who owns it; classification says what may go into a model; retention says how long — including the embeddings and vector stores the model creates.
Data architecture is the prerequisite for AI governance
The AI Governance Framework, model cards and AI registry sit on top of this. See the AI governance practice →
Done, not advised
Data platforms we've designed, migrated and secured.
Migrating customers who are no longer customers
Copying ex-customer data into the new platform creates a fresh processing activity with no lawful basis, extends a retention period you had already exceeded, and doubles the places an erasure request has to reach. If their contract said "delete within 90 days", the migration is the breach. Filter at the model, not after the lift.
Not being able to delete a tenant safely
In a multi-tenant lakehouse, one customer's data sits in shared tables, shared vector stores and shared fine-tunes. "Delete customer X" must reach every derived copy, embedding and backup without touching customer Y. If tenant ID isn't on every object and deletion wasn't designed as a first-class operation, the honest answer to an erasure request is "we can't" — a reportable position. Design deletion before you design ingestion.
“Six to eight weeks of design saves eighteen months of unpicking. Every lakehouse we've been called into late would have cost less to design than it cost to rescue.”
— On sequencing
When we leave
What we leave behind: not a deck, a capability with five working parts.
An engagement ends when the organisation can run these five functions without us. Each one has a named owner, a cadence and a first decision already taken.
Data architecture
Data management driven from the enterprise view, in step with enterprise architecture and security architecture — one set of drawings.
Capability governance
A design authority: the mechanism for making data decisions and guiding action, with the principles to decide against.
Organisation model
Roles and responsibilities for data: owners, stewards, custodians — per domain, by name, not by committee.
Processes
Standard processes for designing, deploying, monitoring and maintaining business data — including quality, lifecycle and deletion.
Change management
A change of mindset about the value of data, through communication, training and awareness — because the business has to own it.
How we engage
Three shapes, depending on where you are.
Data platform review
For an existing lakehouse: the six failure modes assessed, the five maturity components scored, the missing artefacts named, Unity Catalog and security posture reviewed, cost drivers identified, and a risk-ranked roadmap. Fixed price.
Target data architecture
Domain and data model, glossary, lakehouse and catalog design, governance and ownership model — the blueprint, with the security architecture on the same page. Built on our accelerators, so six to eight weeks is realistic rather than optimistic.
Design authority
A fractional data and security design authority: reviewing what's built against the architecture, keeping the standards honest, and training the team to run it without us.
Already built it? Not too late. Not built it yet? Ideal.
Tell us which platform, what's on it, and whether three people would give the same answer about revenue. We'll give you a straight view.