Fail-closed exports: when your report should refuse to run

Most reports are built to succeed. You query some tables, sum some columns, write a file, return a download link. If a number is wrong, nobody finds out until a customer, or an auditor, does.

For a commute-emissions platform whose exports go to external auditors, that wasn’t acceptable. The client needed to know that if an export was produced, its totals were right. So we built it the other way around: the export has to prove the numbers agree before it’s allowed to exist.

The shape of the problem

The system logs individual commute trips and maintains pre-aggregated daily rollups per user, site and organisation, across dimensions like emissions, transport mode and data quality. Rollups make dashboards fast. They also create a second source of truth.

Over time, any of these will happen:

• A bug in the rollup write path skips a dimension.
• A backfill updates raw trips but not the aggregates.
• A concurrent write races and one update is lost.

In each case the dashboard shows one number and the raw data would produce another. An export that reads from the rollups would ship the wrong number with full confidence.

Rule 1: the export re-derives totals independently

The export pipeline does not reuse the aggregation code that produces rollups. It has its own, deliberately simple, derivation that reads raw trip records and sums them per day:

def derive_daily_totals(trips: Iterable[Trip]) -> dict[date, Totals]:
    totals: dict[date, Totals] = defaultdict(Totals)
    for trip in trips:
        day = trip.started_at.astimezone(ORG_TZ).date()
        totals[day].emissions_g += trip.emissions_g
        totals[day].trips += 1
        totals[day].by_mode[trip.mode] += trip.emissions_g
    return totals

Then it loads the stored rollups for the same range and compares:

def reconcile(derived, stored, tolerance_g=0) -> list[Divergence]:
    issues = []
    for day in sorted(set(derived) | set(stored)):
        d, s = derived.get(day), stored.get(day)
        if d is None or s is None or abs(d.emissions_g - s.emissions_g) > tolerance_g:
            issues.append(Divergence(day=day, derived=d, stored=s))
    return issues

If issues is non-empty, the export stops. It doesn’t pick the “better” number. It doesn’t write a partial file. It records the divergence, alerts us, and returns an error to the user that says the data needs attention.

Sharing the aggregation logic between the write path and the check would defeat the purpose. If the same function has the same bug in both places, they’ll agree and both be wrong. The check must be an independent implementation, even if that feels like duplicated code.

Rule 2: the package proves what it contains

When reconciliation passes, the export produces a ZIP of 13 files: per-entity CSVs, a method note, the reconciliation report itself, and a manifest. Every file is SHA-256 hashed and the hashes are listed in the manifest:

{
  "generated_at": "2026-09-30T02:14:11Z",
  "range": { "from": "2026-07-01", "to": "2026-09-30" },
  "files": {
    "trips.csv": "9f86d081884c7d659a2feaa0c55ad015...",
    "daily_totals.csv": "2c26b46b68ffc68ff99b453c1d304134...",
    "reconciliation.json": "ba7816bf8f01cfea414140de5dae2223..."
  }
}

An auditor can verify that nothing changed between generation and delivery. The reconciliation report is inside the package, so the evidence that the totals matched travels with the totals.

Rule 3: long jobs don’t fight the gateway

An export over a quarter of data for a large organisation takes longer than API Gateway’s 29-second synchronous limit. The usual answers are a queue and a worker, or Step Functions. Both are fine, but for one job type they’re more moving parts than needed.

We used a self-invoking Lambda:

def handler(event, context):
    if event.get("phase") != "run":
        job = ExportJob.create(status="queued", params=event["params"])
        lambda_client.invoke(
            FunctionName=context.function_name,
            InvocationType="Event",  # async
            Payload=json.dumps({"phase": "run", "job_id": job.id}),
        )
        return {"statusCode": 202, "body": {"job_id": job.id}}

    job = ExportJob.get(event["job_id"])
    run_export(job)  # reconcile, build ZIP, upload to S3

The first invocation returns a job ID in milliseconds. The second runs with the full 15-minute Lambda budget. The client polls a status endpoint, and when the job is done it receives a presigned S3 URL that expires in an hour.

Rule 4: raw records are never deleted

Rollups can be rebuilt. Raw trips cannot. Every correction, cancellation or dispute creates a new record that references the original; nothing is updated in place. That’s what makes the independent re-derivation possible at all. If you overwrite raw data, you’ve lost the ability to check anything.

What this costs

The export is slower than one that reads rollups directly, and the reconciliation code is extra work to maintain. In exchange, the client has never shipped a wrong number to an auditor, and when divergences did occur during development, the export caught them before anyone outside the team saw them.

For systems where someone external relies on your numbers, that trade is worth it every time. If you need an export, ledger or report that has to be right, talk to us.