Metadata as code
Every few weeks someone would find a new kind of metadata we hadn’t planned for. A product hierarchy. A calendar. A relationship between two things that used to be independent. And every time, the answer was the same: new table, new columns, a migration, model changes, serialiser changes, an API change. A week of engineering before anyone could store one new idea.
After enough rounds of that I stopped asking how to model the next concept and asked why metadata was in a relational model at all.
What goes wrong
Enterprise metadata ends up spread across tables, spreadsheets, wiki pages, config files and whatever catalog UI someone bought. The same problems show up every time.
Modelling never ends. The domain keeps changing and the schema has to chase it. Nothing is ever finished, it’s just waiting for the next migration.
Vocabulary drifts. Column percentage, Column % and Col % all mean
the same thing. A free-text column will happily hold all three, and every
consumer has to learn the aliases.
Lineage lives in prose. “Derived from the open-ended version of this question” is useful to a person and useless to a program. A dependency written as a sentence can’t be traversed, validated or drawn.
History is an afterthought. “What did this definition say in March, and who approved it?” You can answer that with audit columns and temporal tables, but only if you built them up front. Most teams didn’t.
Agents need structure. An AI assistant answering questions about your metrics needs explicit relationships and controlled values it can rely on. A human-oriented description isn’t enough.
The idea
Keep the metadata once, as code, in version control. Keep a small database only for the operational things that actually need transactions.
That’s the whole proposal. The rest of this post is what it looks like in practice.
The format we use is OKF, the Open Knowledge Format. An OKF bundle is a
directory of markdown files, one per concept. Each file has frontmatter at
the top with a type key and whatever structured fields you need, then a
normal markdown body for the explanation. Code filters on the frontmatter.
People read the body. It’s the same file, so there’s no export step and no
second copy to fall out of sync.
The metadata has three layers, and all three are OKF documents in the same bundle:
- Schema says what a valid document of each type looks like: required frontmatter fields, types, cardinality, which aspects are allowed.
- Taxonomy is the controlled vocabulary. Metric types, summarisation
types, chart types. The only place
COLUMN_PERCENTAGEis spelled. - Ontology says which relationships are allowed between which kinds of thing.
The ontology part sounds heavier than it is. You don’t need RDF or OWL. A short list of permitted edges is enough:
metric ── belongs_to ──► logic_group
metric ── depends_on ──► metric
metric ── uses_base ───► base
metric ── applies_to ──► product
An individual metric in the bundle then looks something like this:
---
type: metric
id: awareness-total
title: Awareness - Total
summarisation: [ABSOLUTE_COUNT, COLUMN_PERCENTAGE]
visualisation_default: LINE
visualisation_allowed: [LINE, BAR]
belongs_to: [awareness]
depends_on: [awareness-open-end]
uses_base: [asked]
---
# Awareness - Total
Share of respondents who name the brand without being prompted.
Coded from the open-ended answer, so it depends on that metric.
## Visualisation
Line by default, for trends over time. Bar when comparing brands at a
single point in time.
Everything a program needs is in the frontmatter. Everything a person needs,
including the caveats, is in the body. OKF only insists on type. Every
other key is ours, and consumers are expected to keep keys they don’t
recognise instead of rejecting the file.
Every entry gets checked the same way. The schema checks structure, the
taxonomy checks that every value is an approved term, the ontology checks
that every link is a permitted kind, and referential validation checks that
the thing on the other end of the link actually exists. depends_on is no
longer a sentence. It’s an edge you can walk.
Adding a new concept now means adding a type and a few permitted edges. No migration, just new documents in the bundle.
Nobody should have to learn git
This is where people push back, and fairly. The people who own these definitions aren’t engineers. Asking them to open pull requests would be a fast way to kill the whole thing.
So they don’t. They edit in a UI. Behind it, each change gets its own short-lived branch, the UI writes the OKF document, and automated validation runs. Business approval happens in the UI, the same way it did before. Git is an implementation detail.
A change only merges when the approval is done, the schema, taxonomy and ontology checks pass, every reference resolves, and the change still validates against the latest main. If the content changes after it was approved, the approval is reset. People are approving what gets merged, not an earlier version of it.
main is the approved state of the metadata. Always.
Conflicts, in plain language
Two people start from the same version. One change is approved and merged. The other is still waiting for approval. Before the second one merges, it gets revalidated against the new main.
There are two kinds of conflict, and the second is the one that matters.
A technical conflict is when both changes edit the same property. Git catches that.
A semantic conflict is when both changes merge cleanly and the result is still wrong. Change A marks a logic group inactive. Change B adds a new metric that belongs to that logic group. They touch different files, so git sees no conflict. Ontology validation does, and blocks B.
The user never sees merge markers. They see something like: “Awareness was changed while this request was waiting for approval. Review the latest definition before resubmitting.” The message uses their terms, not git’s.
History you don’t have to build
Git already keeps every version. Nothing is overwritten. Any previous commit can be read back.
The only extra piece is a small table that maps releases to commits:
Release 21 → 8ac912
Release 22 → a52be3
Release 23 → e972bc
With that, the service can answer three kinds of request:
GET /metadata/metric/X current approved definition
GET /metadata/metric/X?release=21 a specific release
GET /metadata/metric/X?asOf=2026-03-01 whatever was live on that date
The date resolves to a release, the release resolves to a commit, and the OKF document at that commit comes back with its links. So the relationships and lineage are historical as well, not just the field values. If you ever need to reproduce an analysis from last year, you get the definitions as they were then, including how they were connected.
Each release also packages the full OKF bundle, the metadata plus its own data model, as one immutable archive. Serving different versions to different consumers then works as a lookup order: an explicit request, then a client’s pinned release, then its channel (stable or beta), then the default. We haven’t turned that on yet. The mechanism works, but how long we support old versions and how we deprecate them are policy questions nobody has answered.
Don’t read git on every request
The service shouldn’t be parsing a repository to answer an API call. At deploy time, a compiler step validates the OKF bundle on main and builds a runtime snapshot: compiled JSON, a cache, a search index. The service reads that.
OKF bundle → compile → runtime representation
It works like building source code. Those outputs are generated and thrown away on the next build. Nobody maintains them as a second data model.
So what’s the database for?
Only the operational side: releases and the commit behind each one, approval and workflow state, validation results, deployment status, audit events, maybe in-progress edit sessions. Things that really do need transactions.
The definitions, schemas, taxonomy, ontology and relationships are not in there. That’s deliberate. The moment a copy of the definitions shows up in a table, you have two sources of truth and they will drift.
What it costs
It isn’t free. You’re swapping database constraints for CI checks, and CI only catches what someone thought to check. There’s no ad hoc query layer over the source, so cross-cutting questions go through the compiled snapshot or a search index. Generated artefacts go stale if the build doesn’t run. And the editing UI has to round-trip fields it doesn’t understand, or it will quietly delete them on save. Ask me how I know.
For a corpus like ours, a few hundred governed definitions that change slowly and have to be reviewed, those costs are easy to live with. What we get back is that a new idea in the domain is a new file, not a new table.