How the pipeline works
This dataset is built by the Anu framework — a staged, documented data-construction pipeline. Each script carries a phase prefix (L / P / V / M / A / O) that says what it does, so the code reads as an ordered method, not a pile of scripts.
Pipeline phases
-
L##
Loading
Fetch and read the raw source data — API pulls, archived tables, digitised book figures — and validate that the fetched units match what the named source promises.
-
P##
Processing
Construct each series: rebase, splice vintages, deflate, and combine inputs. Composite and formula series are assembled here, with a dimensional-analysis check whenever units differ. Processing — and processing only.
-
V##
Validation
Check every constructed series against the original published figures and tables — mean absolute error, endpoint sanity, unit and scale audits. A series only passes when it reproduces the source values. The exact figure, table, or page being reproduced is cited in that series' provenance record (Explore → the construction drawer).
-
M##
Manual adjustment
Apply documented hand corrections where the source itself requires them — each adjustment recorded with its rationale, never a silent edit.
-
A##
Analysis
Derive the downstream analytical quantities the book reports (ratios, trends, decompositions) from the validated series.
-
O##
Output
Write the publishable artifacts: the tidy per-series CSVs, the Parquet files, the data dictionary, and the per-series provenance records that ship with the data.
What the series IDs mean
Series IDs are a separate scheme from the pipeline phases above — a letter in a series ID says nothing about how it was built, only what kind of series it is.
Primary series 95
Primary series — the figures and tables from the book or study being replicated. The digits encode the original work's chapter and sequence (e.g. S201 = Chapter 2, series 01); the exact source figure/table/page for each series is cited in its provenance record.
Extra series 15
Extra series — book-appendix series and series drawn from other studies. These are listed after all primary series, split into a book-appendix section and an other-studies section.
Reproduce a figure
Every chart on this site is built by one small, documented transform that turns a published series file into the table the chart plots — then an optional client-side reindex. Below is that exact transform, in both R and Python; pick a language, copy or download it, and you reproduce the figure from the same CSV the Download page serves.
# Pivot a long series CSV (columns: year, value, optional subseries_id) to the
# wide year-by-subseries table the site charts. Faithful port of the app transform.
pivot_series <- function(path) {
df <- read.csv(path, colClasses = "character", check.names = FALSE)
if (!all(c("year", "value") %in% names(df))) stop("need year + value columns")
df$subseries_id <- if ("subseries_id" %in% names(df)) trimws(df$subseries_id) else ""
df$subseries_id[df$subseries_id == ""] <- "value"
df$year <- trimws(df$year)
df$value <- trimws(df$value)
df <- df[nzchar(df$year), , drop = FALSE]
df <- df[!(df$value == "" | tolower(df$value) == "nan"), , drop = FALSE]
subs <- unique(df$subseries_id) # subseries in first-seen order
years <- unique(df$year) # years in first-seen order
wide <- data.frame(year = years, stringsAsFactors = FALSE)
for (s in subs) {
m <- df[df$subseries_id == s, c("year", "value")]
wide[[s]] <- as.numeric(m$value[match(wide$year, m$year)])
}
keep <- rowSums(!is.na(wide[, subs, drop = FALSE])) > 0 # drop all-blank years
wide[keep, , drop = FALSE]
}
tbl <- pivot_series("S201.csv") # one trace per subseries column# Pivot a long series CSV (columns: year, value, optional subseries_id) to the
# wide year-by-subseries table the site charts. Faithful port of the app transform.
import csv
def pivot_series(path):
with open(path, encoding="utf-8", newline="") as f:
reader = csv.DictReader(f)
fields = reader.fieldnames or []
if "year" not in fields or "value" not in fields:
raise ValueError("need year + value columns")
has_sub = "subseries_id" in fields
table, sub_order, year_order = {}, [], []
for rec in reader:
year = (rec.get("year") or "").strip()
val = (rec.get("value") or "").strip()
if not year:
continue
sub = (rec.get("subseries_id") or "").strip() if has_sub else ""
sub = sub or "value"
if sub not in sub_order:
sub_order.append(sub)
if year not in table:
table[year] = {}; year_order.append(year)
if val and val.lower() != "nan":
table[year][sub] = val
header = ["year"] + sub_order
rows = []
for year in year_order: # first-seen year order
row = [year] + [table[year].get(s, "") for s in sub_order]
if any(c != "" for c in row[1:]): # drop all-blank years
rows.append(row)
return header, rows
header, rows = pivot_series("S201.csv") # one trace per subseries column# Reindex a numeric column so the base year = 100 (the chart's index toggle).
reindex_to_100 <- function(years, values, base_year) {
base <- values[match(base_year, years)]
if (is.na(base) || base == 0) stop("base year has no value")
100 * values / base
}# Reindex a numeric column so the base year = 100 (the chart's index toggle).
def reindex_to_100(years, values, base_year):
base = values[years.index(base_year)]
if base in (None, 0):
raise ValueError("base year has no value")
return [100 * v / base if v is not None else None for v in values]The app plumbing around these transforms (the FastAPI routes, the cache build, the Plotly rendering) is Python-only and not ported to R — porting the web server would invent code that doesn't exist. The data transforms above are the part you reproduce a figure with; the full pipeline is on GitHub.