How the pipeline works
This dataset is built by the Anu framework — a staged, documented data-construction pipeline. Each script carries a phase prefix (L / P / V / M / A / O) that says what it does, so the code reads as an ordered method, not a pile of scripts.
Pipeline phases
-
L##
Loading
Fetch and read the raw source data — API pulls, archived tables, digitised book figures — and validate that the fetched units match what the named source promises.
-
P##
Processing
Construct each series: rebase, splice vintages, deflate, and combine inputs. Composite and formula series are assembled here, with a dimensional-analysis check whenever units differ. Processing — and processing only.
-
V##
Validation
Check every constructed series against the original published figures and tables — mean absolute error, endpoint sanity, unit and scale audits. A series only passes when it reproduces the source values. The exact figure, table, or page being reproduced is cited in that series' provenance record (Explore → the construction drawer).
-
M##
Manual adjustment
Apply documented hand corrections where the source itself requires them — each adjustment recorded with its rationale, never a silent edit.
-
A##
Analysis
Derive the downstream analytical quantities the book reports (ratios, trends, decompositions) from the validated series.
-
O##
Output
Write the publishable artifacts: the tidy per-series CSVs, the Parquet files, the data dictionary, and the per-series provenance records that ship with the data.
What the series IDs mean
Series IDs are a separate scheme from the pipeline phases above — a letter in a series ID says nothing about how it was built, only what kind of series it is.
Primary series 33
Primary series — the figures and tables from the book or study being replicated. The digits encode the original work's chapter and sequence (e.g. S201 = Chapter 2, series 01); the exact source figure/table/page for each series is cited in its provenance record.
Extra series 26
Extra series — book-appendix series and series drawn from other studies. These are listed after all primary series, split into a book-appendix section and an other-studies section.
Reproduce a figure
Every chart on this site is built by one small, documented transform that turns a published series file into the table the chart plots — then an optional client-side reindex. Below is that exact transform, in both R and Python; pick a language, copy or download it, and you reproduce the figure from the same CSV the Download page serves.
# Read a wide series CSV (1 metadata row, then a "Year,<sub>,<sub>,..." header)
# into the exact table the site charts. Faithful port of the app transform.
read_series <- function(path, meta_rows = 1L) {
lines <- readLines(path, encoding = "UTF-8")
header <- strsplit(lines[meta_rows + 1L], ",", fixed = TRUE)[[1]]
subs <- trimws(header[-1L]) # subseries column names
data <- lines[(meta_rows + 2L):length(lines)]
rows <- lapply(data, function(ln) {
if (!nzchar(trimws(ln))) return(NULL)
cells <- strsplit(ln, ",", fixed = TRUE)[[1]]
year <- trimws(cells[1])
if (!nzchar(year)) return(NULL)
vals <- vapply(seq_along(subs), function(i) {
v <- if (length(cells) > i) trimws(cells[i + 1L]) else ""
if (tolower(v) == "nan") "" else v # drop NaN -> blank
}, character(1))
if (!any(nzchar(vals))) return(NULL) # skip all-blank years
c(year, vals)
})
rows <- do.call(rbind, Filter(Negate(is.null), rows))
df <- as.data.frame(rows, stringsAsFactors = FALSE)
names(df) <- c("year", subs)
df[] <- lapply(df, function(col) suppressWarnings(as.numeric(col)))
df
}
tbl <- read_series("S506.csv") # one trace per subseries column# Read a wide series CSV (1 metadata row, then a "Year,<sub>,<sub>,..." header)
# into the exact table the site charts. Faithful port of the app transform.
import csv
def read_series(path, meta_rows=1):
with open(path, encoding="utf-8", newline="") as f:
reader = list(csv.reader(f))
raw_header = reader[meta_rows]
subs = [c.strip() for c in raw_header[1:]] # subseries column names
header = ["year"] + subs
rows = []
for r in reader[meta_rows + 1:]:
if not r:
continue
year = (r[0] if r else "").strip()
if not year:
continue
cells = []
for i in range(1, len(subs) + 1):
v = (r[i] if i < len(r) else "").strip()
cells.append("" if v.lower() == "nan" else v) # drop NaN -> blank
if any(c != "" for c in cells): # skip all-blank years
rows.append([year] + cells)
return header, rows
header, rows = read_series("S506.csv") # one trace per subseries column# Reindex a numeric column so the base year = 100 (the chart's index toggle).
reindex_to_100 <- function(years, values, base_year) {
base <- values[match(base_year, years)]
if (is.na(base) || base == 0) stop("base year has no value")
100 * values / base
}# Reindex a numeric column so the base year = 100 (the chart's index toggle).
def reindex_to_100(years, values, base_year):
base = values[years.index(base_year)]
if base in (None, 0):
raise ValueError("base year has no value")
return [100 * v / base if v is not None else None for v in values]The app plumbing around these transforms (the FastAPI routes, the cache build, the Plotly rendering) is Python-only and not ported to R — porting the web server would invent code that doesn't exist. The data transforms above are the part you reproduce a figure with; the full pipeline is on GitHub.