Open data portals

01What a portal actually is

A data portal is a catalogue, not an archive of truth. It lists datasets that a public body has chosen to publish, in the form that body found convenient, at the moment it found convenient. In Ireland the central catalogue at data.gov.ie aggregates records from departments, agencies and local authorities; the Central Statistics Office publishes its own series separately; Eurostat holds comparable European figures with its own definitions.

The practical consequence is that a portal search answers the question "what has been published?" and never the question "what exists?". A reporter who treats the two as equivalent will describe an absence of data as an absence of the underlying activity, which is one of the most common errors in data reporting.

02Reading the dataset description before the data

Every serious download begins with the metadata. The fields worth reading are the publisher, the period covered, the update frequency, the licence, the geographic level, the date of last modification and any note on methodology or revisions. Together they tell you what a column can support.

Two fields deserve particular suspicion. "Update frequency" is an intention, not a record: a dataset marked monthly may have last changed two years ago, and the modification date will show it. "Period covered" often describes the intended range rather than the range with usable values, so a file labelled 2015 to 2025 may hold empty cells for the first three years.

03Licences and what they permit

Most Irish and European public data is released under a permissive licence that allows reuse with attribution, but the exact terms vary between publishers and occasionally between datasets from the same publisher. Before building anything durable on a source, record the licence name, the version and the attribution wording the publisher asks for.

Licence terms matter beyond legal compliance: they determine whether you can republish the file, whether you can distribute a cleaned version, and whether a derived visualisation can be reused by others. Where a licence is absent or ambiguous, the safe course is to ask the publisher in writing and keep the answer.

04Formats and what each one costs you

Formats carry different amounts of structure and different risks. A machine-readable file with a documented schema is the least troublesome. A spreadsheet built for human reading, with merged cells, colour-coded meaning and totals inside the data area, needs careful restructuring before analysis. A table locked inside a PDF has to be extracted, and extraction introduces its own errors that must be checked line by line against the original.

05Coverage gaps are findings, not obstacles

The interesting question in a portal is often which bodies do not appear. If twenty-six local authorities are meant to report the same indicator and nineteen have published, the seven silences are a legitimate line of inquiry: they may reflect a different recording system, a resource problem, a genuine policy difference or an oversight.

Documenting the gap properly means recording exactly what you searched for, when, with which terms, and what response you received when you asked. That record turns "we could not find the data" into a defensible statement about the state of publication.

06Comparability across publishers and years

Two datasets that use the same word rarely measure the same thing. A count of incidents may include or exclude cases later withdrawn; a population denominator may come from a census year or an estimate; a category may have been redefined mid-series. Before any comparison, align definitions, the reference date, the geographic boundaries and the treatment of unknowns.

Boundary changes are a specific trap in Irish local data, where administrative areas have been merged and renamed. A time series that crosses a boundary change is not a time series unless the earlier figures have been restated, and restatement should be documented by the publisher rather than assumed by the analyst.

07Building a source record

Every dataset used in a published story should have a source record containing the download date, the file name and size, a checksum or at least a stored copy of the original file, the URL, the licence, the publisher contact and any correspondence about definitions. Portals change: files are replaced silently, columns are renamed, historical values are revised without notice.

Keeping the original file is not an act of distrust. It is the only way to answer, months later, the question of whether a figure changed because the world changed or because the publisher revised the series.

08The same data in two shapes: files and endpoints

Many catalogues offer a dataset twice, as a downloadable file and as a query endpoint, and the two are not always the same data. A file is a snapshot taken when the publisher last exported it; an endpoint frequently reads a live table and will therefore return different values tomorrow. A story built on an endpoint has no fixed source unless the response is saved at the moment it is used.

Endpoints also impose limits that quietly truncate results. A default page size returns the first few hundred rows and nothing else; a filter applied by default may exclude withdrawn or superseded records; a sort order can change between requests so that pagination misses rows. The safe procedure is to record the full query, the time of retrieval and the row count returned, then reconcile that count against the total the publisher states, and to keep the raw response exactly as received alongside the download it is meant to match.

09What a portal cannot tell you

A portal holds records of administrative processes, and administrative records describe what an organisation noticed and chose to write down. Under-reporting, definitional narrowing and recording delays are properties of the system, not of the phenomenon. A neutral write-up therefore describes what the data records rather than what happened, and states the distance between the two wherever it is known.