Reproducible method

01Why reproducibility is an editorial standard

A published figure is a claim, and a claim that cannot be reconstructed cannot be defended. Reproducibility means that a colleague, given the same sources and the recorded steps, arrives at the same number. It protects against error before publication and against unfounded challenge afterwards.

It is also practical. Stories return: a series is updated, a regulator disputes a figure, a follow-up needs the same table a year later. Work organised for reproduction takes hours to revisit; work held in an undocumented spreadsheet has to be redone from the beginning, if it can be redone at all.

02Separating raw, processed and output

The minimum structure is three separated layers. Raw data is stored exactly as obtained and never modified. Processed data is generated from raw by recorded steps. Outputs — tables, charts, the figures quoted in the story — are generated from processed data.

Because every layer below can be rebuilt, an error found late is fixed at its source and flows through, rather than being patched in three places with the risk that one is missed. The rule that makes this work is simple and absolute: nothing is edited by hand at a layer that something else depends on.

03Scripted steps over manual edits

A script is a record of what was done. A sequence of manual edits is a memory of it. Even a plain list of ordered, precisely described operations is far better than nothing, but code that runs from raw input to final table is the only form that can be verified exactly.

Scripts should be readable rather than clever. Comments explaining why a decision was made are more valuable than comments describing what a line does, because the decisions are where the judgement lies and where a reviewer needs to look.

04Version control and change history

Version control records when a step changed, what changed and why. Without it, the analysis in a published story cannot be distinguished from the analysis as later modified, which matters if a correction becomes necessary.

For teams it also allows review before a figure is published. A change to a calculation can be read by a second person as a change, rather than being invisible inside a file that looks the same as yesterday.

05Documenting the decisions, not only the code

Code shows what was computed; it does not show why one definition was chosen over another. The decision record covers which population was used, how missing values were treated, which outliers were excluded and on what basis, which time period was chosen and why, and which comparisons were considered and rejected.

That last item deserves emphasis. An analysis that tried several specifications and reports the most striking one is misleading unless the search is disclosed. Recording what was tested is a defence against selective reporting, including self-deception.

06Independent review before publication

Every number that appears in a story should be checked by someone who did not produce it, working from the sources rather than from the analyst's summary. Effective review re-derives the headline figure independently, then tests the definitions and the direction of any comparison.

For findings that are contested, technically difficult or legally sensitive, external review by a subject specialist is proportionate. The reviewer's role is not to endorse the story but to test whether the arithmetic and the definitions support the claim being made.

07The small details that break a rebuild

Analyses usually fail to reproduce for mundane reasons rather than conceptual ones, and the same handful of causes recur. A file referenced by a path that exists only on one machine. A spreadsheet that parsed a decimal comma as a thousands separator because the locale of the computer differed. Dates read as month-first on one system and day-first on another. A column of identifiers silently converted to numbers, losing its leading zeros. A calculation whose result depends on the order in which rounding was applied.

Two further causes matter where code is involved. A step that draws a random sample or splits a dataset will produce different output on each run unless the seed is fixed and recorded. And a result can change with the version of the software that produced it, so the version belongs in the record alongside the steps.

None of this requires elaborate infrastructure to avoid. Relative paths, explicit formats for dates and numbers, identifiers held as text, a fixed seed, a recorded software version and one stated rounding rule cover almost all of it, and each of them takes seconds at the moment the step is written rather than a day to diagnose a year later.

08What to publish alongside the story

A methodology note should state the sources with links and access dates, the period covered, the population or unit of analysis, the cleaning and matching rules, the calculations with formulas where relevant, the known limitations and the contact route for corrections.

Where licences allow, publishing the processed dataset and the analysis code is stronger still. Where they do not, the note should say what cannot be released and why, so that the limit is a stated fact rather than an apparent omission.

09Corrections as part of the method

Reproducible work makes corrections cheap and precise: the error is located in a step, the step is fixed, dependent outputs are regenerated, and the correction note can say exactly what changed and how the finding was affected.

The public record should show that history. A dated correction note, kept with the article, is evidence that the method includes the possibility of being wrong, which is the difference between a defensible practice and a confident one.