Case study · Data quality and GIS

Field data, checked before anyone maps it.

A reproducible workflow that takes a raw field collection file, runs 22 explicit checks on every record, sorts what is usable from what is not, and hands the GIS a layer nobody has to take on trust. To prove it works, I broke the data on purpose: 37 errors injected, 37 found.

The use

What it lets you do

Field teams come back with hundreds of records: parcels surveyed, coordinates taken, areas measured, dates written down. Between that file and a map anyone can act on, there is a step that is almost never budgeted for, and it is the one that decides whether the map is worth anything.

This workflow is that step, made explicit and automatic. It reads the raw file, applies the same 22 checks to every record, and returns three things: a GIS layer of the records that passed, a list of the records to send back to the field with the reason attached, and a report of what the collection is worth overall.

Its point is not to clean the data. It is to make the state of the data visible, and to make the same judgement twice in a row on the same file.

The raw file is never modified. The workflow says what is wrong with it, and only lets through what holds.

The problem

Bad records do not look bad

A raw collection file opens cleanly in any spreadsheet. Nothing in it announces itself as wrong. The same identifier used twice, a survey dated after the survey ended, a parcel whose recorded area does not match the polygon drawn around it, a geometry that is empty: read line by line, none of these stand out.

They surface much later, when someone counts parcels twice, or when a total is off and nobody can say why. By then the map has already been shown, and the credibility of the whole dataset goes with it.

So the question the project starts from is not how to correct these records. It is how to know they are there at all, every time, without depending on who happens to be reading the file.

The approach

One rule, one expression, one severity

Every check is written as a single readable expression over the whole table, not as a paragraph in a procedure. A parcel area must be a positive number. A polygon must be topologically valid. A record identifier must be used once. Twenty-two of them: 16 on the attributes, 6 on the geometry.

Each rule that fires writes one row: which record, which rule, what exactly was found, and how serious it is. Nothing is aggregated away, which is what makes the result actionable rather than merely alarming. A field team receives the record identifier and the reason, not a percentage.

Severity is what turns the issues back into a decision. A critical issue makes the record unusable as it stands. A high or medium issue sends it back for a human check. A record with nothing against it is exported. That is the whole arbitration, and it is the same for every record.

The proof

How I know the checks actually work

A quality control that finds nothing is indistinguishable from a quality control that does nothing. So the dataset is generated with its faults planted deliberately, and the generator keeps the list: 37 known errors, of known types, in known records.

The pipeline then runs blind and is scored against that list. It recovered all 37 out of 37, and raised 5 issues beyond them. Those five are not false alarms: when two records share an identifier, both are flagged, and when two parcels overlap, both are named. One fault, two rows.

The whole thing runs from one command, and reproduces every output file byte for byte. That matters more than the score: a control nobody can re-run is an opinion.

The numbers

What the run says about the collection

Across the hundred records, the checks raise 42 issues, which resolve into three groups of records:

The split between the two families of issues is the useful part. The 25 attribute issues are mostly clerical and cheap to fix: a missing collector, a status typed outside the allowed list, a date that cannot exist. The 17 spatial issues are the expensive ones, because they mean someone has to go back to the parcel: geometries outside the study area, polygons that do not match the recorded area, parcels claimed twice.

Read that way, 37 records held back out of 100 is not a verdict on the field teams. It is a work list, sorted by what it will cost to clear.

The GIS

Some errors are only visible on a map

Attribute checks cannot see everything. A parcel can have a plausible identifier, a plausible date and a plausible area, and still sit two hundred kilometres from the study area, or lie exactly on top of its neighbour. That only shows up once the records are drawn.

So the workflow exports three GIS-ready layers rather than one: the validated parcels, the records held back, and the overlap surfaces themselves, computed as geometry. The last one matters because an overlap is not a property of a record, it is a relation between two, and it has to be looked at as a shape before anyone can decide which of the two is wrong.

Opened together in QGIS, the three layers make the review a matter of looking rather than of cross-referencing spreadsheets.

The reporting

The same run, for whoever has to decide

The issue register is for the person who will fix the records. It is the wrong object for the person who has to decide whether the collection can be published, or whether a site needs a second field visit.

So the same run also renders a one-page dashboard: the headline counts, the issues broken down by type and by severity, the status of every record grouped by site, and the control map. It is generated by the pipeline, from the same outputs, so it cannot drift away from the data it describes.

Site Beta is the case worth reading: it holds four of the six unusable records. That is not visible in the totals, and it is the kind of thing that decides where the next field mission goes.

The limits

What this workflow does not do

It does not correct anything. A record sent back for review stays exactly as it arrived, and the decision belongs to the people who collected it. Automatic correction would only move the problem one step further away from the person able to settle it.

It does not check whether a record is true. A parcel can pass all 22 rules and describe a field that was never visited. What the workflow certifies is internal consistency, which is a precondition for trust, not trust itself.

And the rules are tied to a collection form. Change the form, change the study area, change what counts as a plausible area, and the catalogue has to be revisited. It is written to be read and amended, rather than to be general.

Finally, the run shown here is on 100 synthetic records. That size is enough to demonstrate the logic and to verify the detection, and small enough to be read entirely by eye. It says nothing about how the workflow behaves on a collection of a hundred thousand.

To close

A control that can be argued with

The point of writing the rules down, one expression each, is that someone can disagree with them. A field lead can say that a parcel of 0.2 hectares is perfectly normal here, and the rule changes. That conversation is impossible when the checks live in someone's head or in a spreadsheet nobody reopens.

What I take from this project is the order of operations: decide what a valid record is, then measure the data against it. The 37 out of 37 is not the interesting number in itself. It is the evidence that the check does what it claims, which is the only thing that makes the other numbers worth reading.

Method

The detail, for anyone who wants to check