A scan of data quality (DQ) frameworks and tools used across government, plus a read of 27 public GitHub repositories from 14 public sector organisations. It looks at which tools, quality dimensions and standards appear in real code, and what that means for reporting quality on critical data assets (CDAs) and bringing them into the National Data Library (NDL).
Choosing or connecting a catalogue and a quality tool? See Catalogues and quality tools: how 20 catalogue, platform and quality products connect, five integration patterns from bundled suites to standards-first, a lock-in checker and an RFI and RFP question builder.
The strongest tools are domain-specific: the Home Office profiler (169 rule modules on Spark), the DfE open data screener (63 checks), and the planning data pipeline's YAML expectations engine. General frameworks are almost absent from public code.
MHCLG's specification maps 45 issue types to a quality dimension, a severity level and who must fix it (internal or data provider). It also defines a 7-step quality ladder from "no data" to "trustworthy". This is a ready-made pattern for CDA issue logs.
The cross-government metadata exchange model (LinkML, DCAT-based, outputs JSON Schema and SHACL) has no quality properties. A CDA's baseline scores can't be exchanged in a machine-readable way today.
Uniqueness, validity and accuracy show up most in code. Timeliness is the least covered (6 of 21 repos), and none of the 45 planning issue types measures it. The six DAMA UK dimensions are named explicitly only in guidance repos.
The DQ Hub's resources repo is nearly empty and was last updated in 2022. The admin data quality framework stopped in 2023, and the question bank in 2024. Two once-public MoJ validation tools (data_linter, mojap-metadata) can no longer be reached.
Only DfE's screener enforces GSS harmonised ethnicity values, and only NHS England's linkage framework tests for uneven error across population groups. No repo reports completeness of protected-characteristic fields separately.
Limits of this scan. GitHub's code search needs a sign-in, which this session doesn't have, so repos were found by name and description and then read in full. Private repos and commercial platform configurations are invisible. Some tool use inside large repos may be missed.
Every repository read for this analysis. Filter by sector or role. Status is based on the last commit: active means within 6 months, quiet means within 2 years, dormant means older than that.
| Repository | Organisation | Role | What it does | Tools | Dimensions in code | Standards | Last commit | Status |
|---|
Counts come from pattern matching across all text and code files. "Uniqueness" includes any mention of duplicates, so it is the most generous count.
| Type | Examples | Public sector evidence | Fit for CDA reporting |
|---|---|---|---|
| Government frameworks | Government Data Quality Framework (2020), DQAP guide and issues framework (2026), OSR QAAD toolkit and matrix, NHS Data Quality Maturity Index | Widely used. QAAD reports are routine at DWP, NHSBSA and Forestry | Sets the method |
| Open-source frameworks | Great Expectations, Soda Core, Pandera, dbt tests, Deequ (AWS Glue Data Quality), R validate, pointblank | Great Expectations at ONS. R validate comes from Statistics Netherlands and the European Statistical System | Good for a starter kit |
| Commercial platforms | Collibra Data Quality, Experian Aperture Data Studio, Informatica, Microsoft Purview Data Quality, Ataccama | Collibra is listed on G-Cloud. NHSBSA bought a Datactics profiling tool (Contracts Finder, 2023). Deployments are not visible in public code | |
| Home-grown rule engines | Home Office profiler, DfE eesyscreener, MHCLG expectations, PHS dashboards | The strongest public evidence | Reuse patterns |
| Metadata quality scoring | data.europa.eu Metadata Quality Assessment (DCAT-AP, DQV, SHACL) | EU-wide portal scoring. No UK equivalent found | Model for NDL |
| Catalogues of public sector code | GitHub Repos Asset Catalogue: every UK public sector code project on GitHub, by topic, organisation and capability | Open edition public; covers central, devolved and local government code | Find code to reuse |
| International frameworks | US FCSM Framework for Data Quality (2020), ABS Data Quality Framework and quality statements | National statistics offices | Benchmark |
Status as of 14 October 2026. "Built" items are in the open ndl-dq-assurance kit.
| Recommendation | Status | What exists now |
|---|---|---|
| Add quality to the metadata model using DQV | LinkML extension, JSON Schema, SHACL shapes and an example. The proposal to the GDS model owners has not been sent yet. | |
| Shared issue vocabulary from the planning data model | Built | 43 issue types and 8 dimensions in CSV, SKOS and JSON-LD, mapped to MHCLG equivalents and default impact scores |
| DQAP starter kit | Built | dqkit: 15 checks, exact GOV.UK issues-framework scoring, DQAP template rows, DQV output, and export to Great Expectations and R validate, filled-in DQAP workbooks with issue ageing and escalation, draft case studies, and a weekly pipeline. 60 tests pass, and the browser version matches the Python kit in 30 of 30 parity cases. Try it in your browser. |
| Score metadata quality in the NDL | Built and run | All 84 NDL pages assessed: average 50.9%, CDA metadata coverage 46.4%. See the NDL assurance page. |
| Inclusion checks | Built | Representativeness dimension, unknown-value and group-disparity checks, and the NDL breakdowns and UK-nations checks |
| Restart and preserve community code | Messages drafted for the DQ Hub and MoJ. Not sent. |
MHCLG's quality-challenge repo names i.AI as its partner for AI-driven data quality checks on planning data. It is a direct lead for the NDL Kickstarter.
Reusable rule kits and the planning issue vocabulary lower the measurement and issue-logging days for tier B and C assets.
Add DQV, ISO 8000, ISO/IEC 25012, QAAD and the planning quality ladder as concepts, so the crosswalk covers quality as well as metadata.