Skip to main content

Search the Commons

Type at least 2 letters. Use the arrow keys to move through results, Enter to open, Escape to close.

Ask a question instead: answers drawn from across the Commons, with links to the source. Or browse the site map.

Manage and improve · Tooling research · October 2026

How does the public sector check its data quality, and what does its open code show?

A scan of data quality (DQ) frameworks and tools used across government, plus a read of 27 public GitHub repositories from 14 public sector organisations. It looks at which tools, quality dimensions and standards appear in real code, and what that means for reporting quality on critical data assets (CDAs) and bringing them into the National Data Library (NDL).

In short: a plain English summary
  • This page lists tools that check data quality.
  • Many are free.
  • It says which ones government teams use.
27 repos cloned and scanned 21 genuinely about data quality 14 organisations Search method: repo names and descriptions, then full-file scan
Evidence base. This research started from critical data assets and the National Data Library; the findings apply to any organisation choosing tools. To share rules with the tools listed here, use data contracts and exports; to buy well, see buying and suppliers.
1 of 21data quality repos use a general open-source DQ framework (Great Expectations)
0repos use W3C Data Quality Vocabulary, Soda, Pandera, dbt tests or Deequ
0quality properties in the cross-government metadata exchange model
2repos that encode inclusion checks (DfE ethnicity standards, NHS linkage fairness)

Choosing or connecting a catalogue and a quality tool? See Catalogues and quality tools: how 20 catalogue, platform and quality products connect, five integration patterns from bundled suites to standards-first, a lock-in checker and an RFI and RFP question builder.

What the evidence shows

Finding 1

Teams build their own rule engines

The strongest tools are domain-specific: the Home Office profiler (169 rule modules on Spark), the DfE open data screener (63 checks), and the planning data pipeline's YAML expectations engine. General frameworks are almost absent from public code.

Finding 2

Planning data has the most mature model

MHCLG's specification maps 45 issue types to a quality dimension, a severity level and who must fix it (internal or data provider). It also defines a 7-step quality ladder from "no data" to "trustworthy". This is a ready-made pattern for CDA issue logs.

Finding 3

Quality is missing from the metadata model

The cross-government metadata exchange model (LinkML, DCAT-based, outputs JSON Schema and SHACL) has no quality properties. A CDA's baseline scores can't be exchanged in a machine-readable way today.

Finding 4

The dimensions are uneven

Uniqueness, validity and accuracy show up most in code. Timeliness is the least covered (6 of 21 repos), and none of the 45 planning issue types measures it. The six DAMA UK dimensions are named explicitly only in guidance repos.

Finding 5

The central data quality community has gone quiet

The DQ Hub's resources repo is nearly empty and was last updated in 2022. The admin data quality framework stopped in 2023, and the question bank in 2024. Two once-public MoJ validation tools (data_linter, mojap-metadata) can no longer be reached.

Finding 6

Inclusion checks are rare

Only DfE's screener enforces GSS harmonised ethnicity values, and only NHS England's linkage framework tests for uneven error across population groups. No repo reports completeness of protected-characteristic fields separately.

Limits of this scan. GitHub's code search needs a sign-in, which this session doesn't have, so repos were found by name and description and then read in full. Private repos and commercial platform configurations are invisible. Some tool use inside large repos may be missed.

Evidence table

Every repository read for this analysis. Filter by sector or role. Status is based on the last commit: active means within 6 months, quiet means within 2 years, dormant means older than that.

RepositoryOrganisationRoleWhat it doesToolsDimensions in codeStandardsLast commitStatus

Coverage across the 21 data quality repos

Repos mentioning each quality dimension

Repos using each tool or standard

Counts come from pattern matching across all text and code files. "Uniqueness" includes any mention of duplicates, so it is the most generous count.

The wider landscape

TypeExamplesPublic sector evidenceFit for CDA reporting
Government frameworksGovernment Data Quality Framework (2020), DQAP guide and issues framework (2026), OSR QAAD toolkit and matrix, NHS Data Quality Maturity IndexWidely used. QAAD reports are routine at DWP, NHSBSA and ForestrySets the method
Open-source frameworksGreat Expectations, Soda Core, Pandera, dbt tests, Deequ (AWS Glue Data Quality), R validate, pointblankGreat Expectations at ONS. R validate comes from Statistics Netherlands and the European Statistical SystemGood for a starter kit
Commercial platformsCollibra Data Quality, Experian Aperture Data Studio, Informatica, Microsoft Purview Data Quality, AtaccamaCollibra is listed on G-Cloud. NHSBSA bought a Datactics profiling tool (Contracts Finder, 2023). Deployments are not visible in public codeDepartment choice
Home-grown rule enginesHome Office profiler, DfE eesyscreener, MHCLG expectations, PHS dashboardsThe strongest public evidenceReuse patterns
Metadata quality scoringdata.europa.eu Metadata Quality Assessment (DCAT-AP, DQV, SHACL)EU-wide portal scoring. No UK equivalent foundModel for NDL
Catalogues of public sector codeGitHub Repos Asset Catalogue: every UK public sector code project on GitHub, by topic, organisation and capabilityOpen edition public; covers central, devolved and local government codeFind code to reuse
International frameworksUS FCSM Framework for Data Quality (2020), ABS Data Quality Framework and quality statementsNational statistics officesBenchmark

Standards: used, partly used, missing

Used

  • DAMA UK six dimensions, through the Government Data Quality Framework
  • DCAT, in the metadata exchange model and planning specification
  • LinkML to JSON Schema and SHACL (CDDO)
  • GSS harmonised standards (DfE)
  • CSVW (planning data)

Partly used

  • QAAD, in analysis guidance only
  • ESS validation and VTL, through R validate (international)
  • Frictionless table schema, mentioned but not enforced
  • ISO 8000 and ISO/IEC 25012, in 2 repos as references only

Missing

  • W3C Data Quality Vocabulary (DQV), for publishing quality results as metadata
  • A shared issue-type vocabulary across departments
  • Machine-readable DQAP results
  • Representativeness and inclusion measures

What to do next, for CDAs and the NDL

Status as of 14 October 2026. "Built" items are in the open ndl-dq-assurance kit.

RecommendationStatusWhat exists now
Add quality to the metadata model using DQVProposal draftedLinkML extension, JSON Schema, SHACL shapes and an example. The proposal to the GDS model owners has not been sent yet.
Shared issue vocabulary from the planning data modelBuilt43 issue types and 8 dimensions in CSV, SKOS and JSON-LD, mapped to MHCLG equivalents and default impact scores
DQAP starter kitBuiltdqkit: 15 checks, exact GOV.UK issues-framework scoring, DQAP template rows, DQV output, and export to Great Expectations and R validate, filled-in DQAP workbooks with issue ageing and escalation, draft case studies, and a weekly pipeline. 60 tests pass, and the browser version matches the Python kit in 30 of 30 parity cases. Try it in your browser.
Score metadata quality in the NDLBuilt and runAll 84 NDL pages assessed: average 50.9%, CDA metadata coverage 46.4%. See the NDL assurance page.
Inclusion checksBuiltRepresentativeness dimension, unknown-value and group-disparity checks, and the NDL breakdowns and UK-nations checks
Restart and preserve community codeDrafts readyMessages drafted for the DQ Hub and MoJ. Not sent.

Connections to your work

i.AI and planning data

MHCLG's quality-challenge repo names i.AI as its partner for AI-driven data quality checks on planning data. It is a direct lead for the NDL Kickstarter.

CDA effort model

Reusable rule kits and the planning issue vocabulary lower the measurement and issue-logging days for tier B and C assets.

Standards Rosetta

Add DQV, ISO 8000, ISO/IEC 25012, QAAD and the planning quality ladder as concepts, so the crosswalk covers quality as well as metadata.

Sources