How do we know whether our data is actually reliable?

The short answer

“Is our data reliable?” is a fair question with an unhelpful usual answer, because reliable tends to get treated as a feeling. The report looks right, so it must be right. That is not a test. It is a hope.

Here is the short answer. Data is reliable when it passes a defined set of checks that someone owns. Not “looks fine in the chart”, but passes explicit, repeatable tests: is anything missing, is anything duplicated, are the values legal, do the links between tables hold, is the figure recent enough, and do independent sources agree at the level of individual records. Reliability is the score on those checks, written down, rather than an impression.

So the way to know is to make the checks explicit. Pick the handful of data assets your decisions actually depend on, decide what each check should prove, run them, and give every asset an owner. Once the checks exist, “reliable” stops being a debate and becomes a number you can point at.

What this usually looks like

It usually surfaces after a decision has already been made on a number that turned out to be wrong. A total was too high because a load ran twice. A customer count was too low because half the rows arrived a day late. A margin looked healthy because one amount column held two currencies and nobody noticed.

The uncomfortable part is that nobody could have answered the reliability question beforehand, because no one had written down what reliable meant for that data. The dashboard rendered faithfully. The query was valid SQL. The pipeline reported success. Every green light was honest, and the number was still wrong, because none of those lights was actually checking the data itself.

So the organisation runs on trust and the occasional scare. When a figure is challenged, someone spends a day proving it by hand, the fire goes out, and nothing changes until the next surprise. There is no standing definition of “good”, so there is no way to be confident in the quiet stretches between the scares.

Start with these checks

You can get a real answer in an afternoon, without new tooling. The goal is not to test everything. It is to test the few assets that decisions depend on, against a small, fixed set of checks. A data asset here means one specific table or dataset, such as your orders table or your customer master.

Six checks and an owner. That is enough to turn “we think it is fine” into “it passed, and here is who watches it”.

What the result tells you

Each check answers a different question, so a failure tells you where to look and, just as usefully, who should look.

• A completeness failure points at collection or loading: rows dropped, a late feed, a required field the source never sends.
• A uniqueness failure points at a double load, or a join that multiplied rows.
• A validity failure points at the source or the entry rules: a free-text field where a fixed list was assumed, or a date parsed the wrong way.
• A referential integrity failure points at ordering or identity: a child row loaded before its parent, or two systems that disagree on keys.
• A freshness failure points at the pipeline schedule, not the values.
• A reconciliation failure points at a difference of definition, scope or timing between two systems, which is a business decision as much as a technical one.

A suite that passes is not a guarantee that the data is true. It is a guarantee that the specific ways you know data goes wrong have been ruled out. That is a far stronger position than a chart that merely looks plausible, and it is the honest meaning of the word reliable.

What each check actually means

Each check has a precise meaning worth stating, because loose definitions produce checks that never catch anything.

Completeness is whether the data you should have is present. It has two halves: no missing values in fields that must be filled, and no missing rows against an expected count. A table can be full of valid values and still be badly incomplete when a third of the day’s transactions never arrived.

Uniqueness is the absence of unintended duplicates, so that each real-world thing appears once. You enforce it against a business key, the field that identifies the thing in the real world, such as an order reference or a customer ID, rather than a system-generated row number, which is unique by construction and proves nothing.

Validity is whether values conform to their type, range and domain. Type: a date column holds real dates. Range: an age sits between 0 and 120. Domain: a status is one of an allowed set, not free text. Validity is cheap to check and catches a surprising share of problems at the door.

Referential integrity is whether every child row has a valid parent key. A child row, such as an order line, references a parent, such as an order; referential integrity holds when no child points at a parent that does not exist. Break it and you get orphaned rows that silently drop out of any query that joins the two tables, quietly understating a total.

Freshness is whether the data is recent enough for the decision it supports. It is measured against the decision, not the clock: monthly board data refreshed weekly is fresh, while a live operational figure that is a day old may be stale. The check is the age of the newest record against an agreed limit.

Reconciliation is whether two independent sources agree at the record level, not just on the total. Matching totals can hide two errors that cancel out, and differing totals can be completely correct for a definitional reason. Only lining up the underlying rows tells you which. This is the same discipline behind why numbers disagree across systems: match records, classify every difference, and stop only when each difference has a reason.

Three further ideas separate a basic check suite from a reliable one.

Distribution drift is when the shape of a metric changes unexpectedly, even though every value is individually valid. Average order value halves overnight, or the split of orders across regions shifts hard, with no business reason. Nothing is illegal, so type and range checks stay green; only watching the distribution over time catches it. Drift is the check that finds the problems you did not think to write a rule for.

Business-rule tests encode logic that is true for your organisation specifically: an invoice total equals the sum of its lines, a subscription cannot be active with no plan, a refund never exceeds the original charge. These catch errors that are perfectly valid in the abstract but impossible in your business.

Observability, sometimes called monitoring, is running all of the above continuously and recording the results, so you can see the health of an asset over time and be alerted the moment a check fails, rather than finding out in a meeting. A single pass tells you the data is fine now. Observability tells you it has stayed fine, and raises the alarm when it stops.

The checks themselves are often simpler than they sound. A good pattern is the assertion query: a check written so that it returns rows only when something is wrong. No rows means the check passed. This makes the suite easy to automate, because any output at all is a failure to investigate.

-- Each query returns rows ONLY when the check fails.
-- Empty result = passed.

-- 1. Completeness / validity: a required key must never be null
select order_id
from   orders
where  customer_id is null;

-- 2. Uniqueness: a business key must not repeat
select order_ref, count(*) as rows_found
from   orders
group  by order_ref
having count(*) > 1;

-- 3. Referential integrity: every order must have a real customer
select o.order_id
from   orders o
left join customers c on c.customer_id = o.customer_id
where  c.customer_id is null;

The first query lists any order missing a customer, a completeness and validity failure in one. The second groups orders by their business reference and keeps only references that appear more than once, so every returned row is a duplicate. The third joins orders to customers and keeps the orders where no customer matched, which are the orphaned rows that break referential integrity. Run against clean data, all three return nothing. The day one of them returns rows, you have found a real problem before it reaches a decision.

Finally, ownership. Every asset needs one named person who is accountable for its checks: for defining what good means, for the response when a check fails, and for the fix. Checks without an owner rot. They break, everyone assumes someone else is watching, and the suite quietly becomes decoration.

What good looks like

Good does not mean the data is perfect. It means you can prove its state at any time, and you learn about problems before your stakeholders do. In practice that is five things.

• The checks run automatically on a schedule, next to the data, rather than by hand when someone gets nervous.
• They fail loudly. A failure raises an alert and, where the risk is high, holds the data back rather than publishing it.
• The tests live with the model and are versioned like code, so a change to a definition changes its test in the same place.
• Failures land in an exception process: a queue of records to review and fix, with someone responsible for clearing it.
• Every critical asset has one owner, and the health of each asset is visible over time.

When these hold, the reliability question answers itself. You no longer defend a number in a meeting. You point at its checks.

Common ways this goes wrong

• Judging reliability by eye. A dashboard that looks right is the single most common reason a wrong number survives.
• Reconciling on totals alone. Totals can match by accident and differ for good reasons; only records settle it.
• Writing checks so loose they never fail. A check that has never once failed is usually not proving anything.
• Ignoring freshness. Every value can be correct and the figure still wrong because it is a day old.
• Treating all drift as an error, or none of it. Some distribution shifts are real business change and some are broken pipelines. The check flags the shift; a person decides which it is.
• Running checks with no owner. Unowned checks break silently and the suite becomes theatre.

When reliability needs to be a system

The afternoon of checks is worth doing, and for a small, stable setup it can be enough on its own. It stops being enough when reliability has to be continuous, automated and owned across many assets rather than proven by hand now and then. At that point you are not answering a question. You are running data quality as a system, and it needs to be built like one.

If several checks are failing at once and it is hard to know where to start, that is a sequencing problem in its own right, and worth reading alongside what to fix first when it is all messy.

A decision guide

Each dimension maps to one check and one clear meaning when it fails. Use this as a quick lookup while you build the suite.

Work down the guide once and you will know, with evidence rather than hope, whether your data is reliable and exactly which check to trust it against.