lgtm

pipeline/transform/crime_spatial.py

@@ -2,22 +2,26 @@
 
 Instead of attributing each incident to its published LSOA code, this transform
 counts the anonymised incident *points* that fall within ``buffer_m`` (default
-100m) of each postcode's boundary polygon (the polygon buffered outward). A point
+50m) of each postcode's boundary polygon (the polygon buffered outward). A point
 inside several overlapping buffers counts for each postcode -- the same
 multiplicity the tree-density filter uses for features near more than one
-postcode. The wide 100m buffer deliberately smooths police.uk's snap-to-grid
+postcode. The 50m buffer deliberately smooths police.uk's snap-to-grid
 coordinates, which would otherwise make the count hypersensitive to which side of
 a narrow line a shared "map point" anchor happened to land on.
 
-Counts are **area-normalised**: each postcode's count is divided by its buffered
-catchment area and rescaled by the median catchment area, so the metric reflects
-crime *density* rather than how much ground the buffer sweeps (a median-sized
-catchment is left unchanged; a large rural postcode is no longer inflated simply
-for covering more of the map). Normalising by the buffered area -- the region
-that actually collects points -- rather than the raw polygon keeps tiny unit
-postcodes from being over-inflated by the fixed buffer-ring floor. NOTE: this is
-an incident *density of the surrounding streets*, not a per-resident risk --
-zero-resident commercial centres (Soho, retail parks) legitimately rank high.
+One figure is produced for every postcode and crime type, averaged over the
+**last 7 years** and the **last 2 years**:
+
+* the **average annual incident count** -- ``sum(counts in covered window) * 12 /
+  covered_months`` -- the raw, absolute number of recorded incidents per year,
+  with no per-area or per-capita normalisation. A covered postcode with no
+  incidents of a type gets ``0``; a postcode whose force never published in the
+  window, or whose geometry is unusable, gets a *null* (unknown, never a zero).
+
+This figure is what the server exposes as the filterable crime feature -- the
+headline metric in the right pane. (An earlier version divided it by an ambient
+daytime population to get a per-1,000-people rate; that was hard to read, so the
+absolute per-year count is now used directly.)
 
 **Force-coverage calendar.** police.uk has multi-year publication gaps for whole
 forces (Greater Manchester has published nothing between 2019-07 and the present
@@ -29,28 +33,25 @@ computed against the months the postcode's own force actually published:
   matched it (BTP, which reports nationwide, is excluded from the vote);
   postcodes with no incidents inherit their outcode's majority force, then the
   national modal force.
-* The headline ``"{type} (avg/yr)"`` is the POOLED annualised rate over the
-  force's covered months: ``sum(counts in covered years) * 12 / covered_months``.
-  Years in which the force published nothing contribute neither incidents nor
-  months, so a coverage gap no longer reads as a low-crime period. (Pooling over
-  covered months also fixes the old "divide by years-with-incidents" headline,
-  which inflated sporadic categories by up to ~15x.)
-* The by-year series only emits bars for years with at least
-  ``min_bar_months`` covered months (default 6): annualising a single observed
-  month x12 produced misleading spikes. Each bar is scaled by the force's
-  covered months in that year, not the global month calendar.
+* A window's average pools the counts over the force's *covered* months inside
+  that window and annualises by those months, so a coverage gap shrinks the data
+  rather than reading as a low-crime dip.
+* The by-year series only emits bars for years with at least ``min_bar_months``
+  covered months (default 6).
 * ``covered_years`` (list[struct{year, months}]) is written for every postcode
-  so the server can tell "covered, zero crime" (year listed, no bar) from "no
-  data" (year absent) instead of charting gaps as zeros.
+  so the server can tell "covered, zero crime" from "no data".
 * Postcodes whose boundary buffer is unusable (broken geometry) get null
-  headline columns and an empty ``covered_years`` -- unknown, not zero.
+  figures and an empty ``covered_years`` -- unknown, not zero.
 
-Outputs mirror the old LSOA transform's shape but are keyed on ``postcode``:
+Outputs, all keyed on ``postcode``:
 
-* ``crime_by_postcode.parquet`` -- ``postcode`` + ``"{type} (avg/yr)"`` columns.
-* ``crime_by_postcode_by_year.parquet`` -- one row per postcode: ``postcode`` +
-  ``covered_years`` + nested ``"{type} (by year)"`` ``list[struct{year, count}]``
-  columns, with Serious/Minor rollups.
+* ``crime_by_postcode.parquet`` -- ``"{type} (/yr, 7y)"`` / ``"{type} (/yr, 2y)"``
+  average-annual-count columns (the filterable crime features).
+* ``crime_by_postcode_by_year.parquet`` -- ``covered_years`` + nested
+  ``"{type} (by year)"`` ``list[struct{year, count}]`` per-year raw counts.
+* ``crime_records.parquet`` -- one row per counted incident over the last 7
+  years (``postcode`` + month/type/location/outcome/lat/lon), sorted by
+  postcode so the server can slice each postcode's incidents directly.
 
 Caveat: police.uk coordinates are snapped to a fixed set of anonymous "map
 points", not true locations, and a share of rows have no coordinate at all
@@ -63,6 +64,7 @@ from __future__ import annotations
 import argparse
 import re
 import sys
+import tempfile
 from pathlib import Path
 
 import numpy as np
@@ -80,22 +82,36 @@ from pipeline.transform.postcode_boundaries.loader import load_postcode_polygons
 
 # Serious types first so column order is stable and self-documenting.
 ALL_CRIME_TYPES: tuple[str, ...] = SERIOUS_CRIME_TYPES + MINOR_CRIME_TYPES
+# Output type axis = the 14 leaf types plus the two rollups, in that order.
+ROLLUP_TYPES: tuple[str, ...] = ("Serious crime", "Minor crime")
+ALL_OUTPUT_TYPES: tuple[str, ...] = ALL_CRIME_TYPES + ROLLUP_TYPES
 
-DEFAULT_BUFFER_M = 100.0
+DEFAULT_BUFFER_M = 50.0
 MONTH_DIR_RE = re.compile(r"^\d{4}-\d{2}$")
 STREET_CSV_NAME_RE = re.compile(r"^(\d{4}-\d{2})-(.+)-street\.csv$")
 
+# Trailing-window definitions, in (label, years) form. Each window's average is
+# pooled over the force's covered months inside the window; one average-annual-
+# count column (`(/yr, <label>)`) — the filterable crime feature — is emitted per
+# window.
+WINDOWS: tuple[tuple[str, int], ...] = (("7y", 7), ("2y", 2))
+
+# Per-incident records cover the longest window's calendar years so the
+# "individual crimes" list reconciles exactly with the headline counts: every
+# record is an incident inside that window and vice versa. The headline counts
+# are pooled per calendar year, so the records window must be calendar-year
+# aligned too -- a rolling month span (e.g. a fixed 84 months) would include
+# incidents from a year the headline excludes when the latest month is mid-year.
+RECORDS_WINDOW_YEARS = max(win_years for _, win_years in WINDOWS)
+
 # Minimum covered months for a year to get a by-year chart bar (and to be
 # listed in `covered_years`). Annualising fewer observed months (x12 from a
-# single month at the worst) produces bars dominated by noise, and the first
-# (2010: one month) and current partial year would otherwise always chart as
-# spikes/dips. Six months keeps the annualisation factor <= 2.
+# single month at the worst) produces bars dominated by noise. Six months keeps
+# the annualisation factor <= 2.
 MIN_BAR_MONTHS = 6
 
 # Forces that report nationwide rather than policing a territory. They never
-# define a postcode's home force (their publication calendar says nothing about
-# whether the *territorial* force covering the postcode published), but their
-# incidents still count toward whichever postcodes they fall in.
+# define a postcode's home force, but their incidents still count.
 NON_TERRITORIAL_FORCES = frozenset({"btp"})
 
 COVERAGE_COLUMN = "covered_years"
@@ -106,8 +122,14 @@ LON_BOUNDS = (-9.5, 2.5)
 LAT_BOUNDS = (49.0, 61.5)
 
 # Read CSVs in chunks of files to bound peak memory while keeping the STRtree
-# query vectorised over a useful number of points.
-_CSV_BATCH = 64
+# query vectorised over a useful number of points. Kept modest because the
+# in-window batches also materialise the per-incident record strings.
+_CSV_BATCH = 32
+
+
+def crime_column(type_name: str, window: str) -> str:
+    """Filterable average-annual-count column, e.g. ``"Burglary (/yr, 7y)"``."""
+    return f"{type_name} (/yr, {window})"
 
 
 def _force_calendar(
@@ -115,12 +137,9 @@ def _force_calendar(
 ) -> tuple[list[int], list[str], np.ndarray]:
     """Derive the per-force publication calendar from the CSV paths.
 
-    Each police.uk file lives under ``{crime_dir}/{YYYY-MM}/{YYYY-MM}-{force}-
-    street.csv`` and holds that force's incidents for that month, so file
-    presence IS the coverage signal: a (force, month) with no file published
-    nothing. Returns the sorted distinct years, the force slugs (sorted), and
-    ``months_in_year_force`` of shape (n_forces, n_years) -- how many months
-    each force published in each year.
+    File presence IS the coverage signal: a (force, month) with no file
+    published nothing. Returns the sorted distinct years, the force slugs
+    (sorted), and ``months_in_year_force`` of shape (n_forces, n_years).
     """
     month_force: set[tuple[str, str]] = set()
     for path in csvs:
@@ -142,9 +161,6 @@ def _force_calendar(
     for month, force in month_force:
         months_in_year_force[force_to_idx[force], year_to_idx[int(month[:4])]] += 1
 
-    # Surface coverage gaps loudly: any territorial force missing months inside
-    # the global publication window is exactly the data hole the coverage
-    # masking exists for.
     all_months = {month for month, _ in month_force}
     for force in forces:
         published = {m for m, f in month_force if f == force}
@@ -159,22 +175,25 @@ def _force_calendar(
 
 def _build_tree(
     polygons: np.ndarray, buffer_m: float
-) -> tuple[np.ndarray, shapely.STRtree]:
+) -> tuple[shapely.STRtree, np.ndarray]:
     """Buffer postcode polygons outward by ``buffer_m`` and index them.
 
-    Buffer index == postcode index. Geometries that fail to buffer are replaced
-    with an empty polygon so the index stays aligned; they simply never match.
+    Buffer index == postcode index. Returns the STRtree and a ``usable`` boolean
+    mask: geometries that fail to buffer are replaced with an empty polygon (so
+    the index stays aligned and they never match) and marked unusable, so their
+    crime picture is reported as unknown rather than zero.
     """
     buffers = shapely.buffer(polygons, buffer_m, quad_segs=8)
     broken = shapely.is_missing(buffers) | ~shapely.is_valid(buffers)
     if broken.any():
         print(f"  {int(broken.sum()):,} postcode buffers unusable; left empty")
         buffers[broken] = shapely.from_wkt("POLYGON EMPTY")
-    return buffers, shapely.STRtree(buffers)
+    return shapely.STRtree(buffers), ~broken
 
 
 def _accumulate_counts(
     csvs: list[Path],
+    postcodes: np.ndarray,
     tree: shapely.STRtree,
     type_to_idx: dict[str, int],
     year_to_idx: dict[int, int],
@@ -182,35 +201,49 @@ def _accumulate_counts(
     transformer: Transformer,
     counts: np.ndarray,
     force_votes: np.ndarray,
+    records_shard_dir: Path | None,
+    records_min_ym: int,
 ) -> None:
     """Stream the crime CSVs, counting points-in-buffer per (postcode, type, year).
 
-    Also accumulates ``force_votes`` (n_postcodes, n_forces): how many matched
-    incidents each force's files contributed to each postcode, which later
-    elects the postcode's home force for the coverage calendar.
+    Also accumulates ``force_votes`` (n_postcodes, n_forces) for the home-force
+    election and, when ``records_shard_dir`` is set, writes one parquet shard per
+    batch holding every counted (incident, postcode) pair whose month is within
+    the records window (month index >= ``records_min_ym``).
     """
+    # Type overrides only for the columns we ever read; LSOA is not stored.
     schema = {
         "Longitude": pl.Float64,
         "Latitude": pl.Float64,
         "Month": pl.Utf8,
         "Crime type": pl.Utf8,
+        "Location": pl.Utf8,
+        "Last outcome category": pl.Utf8,
     }
     years = list(year_to_idx)
     total_points = 0
     total_matches = 0
     total_dropped = 0
+    total_records = 0
     unknown_type_counts: dict[str, int] = {}
 
     for start in range(0, len(csvs), _CSV_BATCH):
         batch = csvs[start : start + _CSV_BATCH]
-        # The source file identifies the publishing force (police.uk has no
-        # force column with consistent naming); map each path back to its
-        # force index for the home-force vote.
         path_to_fidx = {}
+        batch_max_ym = -1
         for path in batch:
             m = STREET_CSV_NAME_RE.fullmatch(path.name)
-            if m is not None and m.group(2) in force_to_idx:
-                path_to_fidx[str(path)] = force_to_idx[m.group(2)]
+            if m is not None:
+                ym = m.group(1)
+                batch_max_ym = max(batch_max_ym, int(ym[:4]) * 12 + int(ym[5:7]) - 1)
+                if m.group(2) in force_to_idx:
+                    path_to_fidx[str(path)] = force_to_idx[m.group(2)]
+        # The per-incident record strings (Location, outcome) are by far the
+        # heaviest columns; read them only for batches that fall inside the
+        # records window, so the ~50% of pre-window months cost nothing extra.
+        want_records = records_shard_dir is not None and batch_max_ym >= records_min_ym
+        record_cols = ["Location", "Last outcome category"] if want_records else []
+
         frame = (
             pl.scan_csv(
                 batch,
@@ -218,12 +251,12 @@ def _accumulate_counts(
                 ignore_errors=True,
                 include_file_paths="_source_path",
             )
-            .select("Longitude", "Latitude", "Month", "Crime type", "_source_path")
-            # strict=False: a single malformed Month drops only that row instead
-            # of aborting the whole build (a non-numeric year becomes null and is
-            # filtered out by the year membership check below).
+            .select(
+                "Longitude", "Latitude", "Month", "Crime type", *record_cols, "_source_path"
+            )
             .with_columns(
-                pl.col("Month").str.slice(0, 4).cast(pl.Int32, strict=False).alias("year")
+                pl.col("Month").str.slice(0, 4).cast(pl.Int32, strict=False).alias("year"),
+                pl.col("Month").str.slice(5, 2).cast(pl.Int32, strict=False).alias("_mm"),
             )
             .filter(
                 pl.col("Longitude").is_not_null()
@@ -234,14 +267,11 @@ def _accumulate_counts(
                 & (pl.col("Crime type") != "")
                 & pl.col("year").is_in(years)
             )
-            # Canonicalise legacy pre-2014 crime-type names ("Violent crime",
-            # "Public disorder and weapons") to their current equivalents before
-            # indexing, so ~1.9M historical incidents are counted instead of
-            # dropped. `.replace` leaves current types unchanged.
+            # year*12 + (month-1): an integer month index for window filtering.
+            .with_columns(
+                (pl.col("year") * 12 + (pl.col("_mm").fill_null(1) - 1)).alias("month_index")
+            )
             .with_columns(pl.col("Crime type").replace(LEGACY_CRIME_TYPE_ALIASES))
-            # Map crime types to indices with default=None so an unrecognised
-            # type yields a null index we can *report* rather than silently drop
-            # (the legacy LSOA path surfaced unknown types via its dynamic pivot).
             .with_columns(
                 pl.col("Crime type")
                 .replace_strict(type_to_idx, default=None, return_dtype=pl.Int32)
@@ -253,7 +283,16 @@ def _accumulate_counts(
                 .replace_strict(path_to_fidx, default=-1, return_dtype=pl.Int32)
                 .alias("fidx"),
             )
-            .select("Longitude", "Latitude", "Crime type", "tidx", "yidx", "fidx")
+            .select(
+                "Longitude",
+                "Latitude",
+                "Crime type",
+                *record_cols,
+                "month_index",
+                "tidx",
+                "yidx",
+                "fidx",
+            )
             .collect(engine="streaming")
         )
 
@@ -273,19 +312,15 @@ def _accumulate_counts(
         tidx = frame["tidx"].to_numpy()
         yidx = frame["yidx"].to_numpy()
         fidx = frame["fidx"].to_numpy()
+        month_index = frame["month_index"].to_numpy()
 
         x, y = transformer.transform(lon, lat)
         finite = np.isfinite(x) & np.isfinite(y)
         total_dropped += int((~finite).sum())
         if not finite.any():
             continue
-        x, y, tidx, yidx, fidx = (
-            x[finite],
-            y[finite],
-            tidx[finite],
-            yidx[finite],
-            fidx[finite],
-        )
+        x, y = x[finite], y[finite]
+        tidx, yidx, fidx = tidx[finite], yidx[finite], fidx[finite]
         total_points += x.size
 
         points = shapely.points(x, y)
@@ -306,9 +341,26 @@ def _accumulate_counts(
                 )
             total_matches += point_index.size
 
+            if want_records:
+                total_records += _write_record_shard(
+                    records_shard_dir,
+                    start,
+                    postcodes,
+                    point_index,
+                    postcode_index,
+                    month_index[finite],
+                    frame["Crime type"].to_numpy()[finite],
+                    frame["Location"].to_numpy()[finite],
+                    frame["Last outcome category"].to_numpy()[finite],
+                    lon[finite],
+                    lat[finite],
+                    records_min_ym,
+                )
+
         print(
             f"  files {start + len(batch):,}/{len(csvs):,}: "
             f"{total_points:,} located points, {total_matches:,} postcode matches"
+            + (f", {total_records:,} records" if records_shard_dir is not None else "")
         )
 
     if total_dropped:
@@ -329,19 +381,65 @@ def _accumulate_counts(
         )
 
 
+def _write_record_shard(
+    records_shard_dir: Path,
+    start: int,
+    postcodes: np.ndarray,
+    point_index: np.ndarray,
+    postcode_index: np.ndarray,
+    month_index: np.ndarray,
+    crime_type: np.ndarray,
+    location: np.ndarray,
+    outcome: np.ndarray,
+    lon: np.ndarray,
+    lat: np.ndarray,
+    records_min_ym: int,
+) -> int:
+    """Write one parquet shard of (incident, postcode) records for this batch.
+
+    Each matched pair becomes a row -- the same multiplicity as the count -- so a
+    postcode's records are exactly the incidents that made up its counts. Only
+    incidents within the records window (month index >= ``records_min_ym``) are
+    kept. Returns the number of rows written.
+    """
+    mi = month_index[point_index]
+    keep = mi >= records_min_ym
+    if not keep.any():
+        return 0
+    pidx = point_index[keep]
+    # Build the string columns from Python lists (.tolist()) rather than numpy
+    # object arrays: an all-null Location/outcome slice is an all-None object
+    # array that polars cannot cast to String, whereas a list of str|None infers
+    # a nullable String column cleanly.
+    shard = pl.DataFrame(
+        {
+            "postcode": postcodes[postcode_index[keep]].astype(str),
+            "month_index": mi[keep].astype(np.int32),
+            "crime_type": crime_type[pidx].astype(str),
+            "location": location[pidx].tolist(),
+            "outcome": outcome[pidx].tolist(),
+            "lat": lat[pidx].astype(np.float32),
+            "lon": lon[pidx].astype(np.float32),
+        },
+        schema_overrides={
+            "postcode": pl.String,
+            "crime_type": pl.String,
+            "location": pl.String,
+            "outcome": pl.String,
+        },
+    )
+    shard.write_parquet(
+        records_shard_dir / f"{start:08d}.parquet", compression="zstd"
+    )
+    return shard.height
+
+
 def _assign_home_force(
     postcodes: np.ndarray,
     force_votes: np.ndarray,
     forces: list[str],
 ) -> np.ndarray:
-    """Elect each postcode's home (territorial) force.
-
-    Majority vote of matched incidents per publishing force; non-territorial
-    forces (BTP) are excluded from the vote because their calendar says nothing
-    about local coverage. Postcodes with no votes (no incidents ever, or
-    BTP-only) inherit the majority force of their outcode, then the national
-    modal force, so every postcode gets a coverage calendar.
-    """
+    """Elect each postcode's home (territorial) force by majority incident vote."""
     votes = force_votes.astype(np.int64, copy=True)
     for idx, force in enumerate(forces):
         if force in NON_TERRITORIAL_FORCES:
@@ -354,18 +452,15 @@ def _assign_home_force(
     if not has_vote.any():
         raise ValueError("No incidents matched any postcode; cannot assign forces")
 
-    # Outcode-majority fallback for postcodes with no (territorial) incidents.
     outcodes = np.array([pc.split(" ")[0] for pc in postcodes], dtype=object)
-    national_modal = int(
-        np.bincount(home[has_vote], minlength=len(forces)).argmax()
-    )
+    national_modal = int(np.bincount(home[has_vote], minlength=len(forces)).argmax())
     if (~has_vote).any():
         outcode_modal: dict[str, int] = {}
         voted_outcodes = outcodes[has_vote]
         voted_home = home[has_vote]
         for oc in np.unique(voted_outcodes):
-            counts = np.bincount(voted_home[voted_outcodes == oc], minlength=len(forces))
-            outcode_modal[oc] = int(counts.argmax())
+            tally = np.bincount(voted_home[voted_outcodes == oc], minlength=len(forces))
+            outcode_modal[oc] = int(tally.argmax())
         fallback = np.array(
             [outcode_modal.get(oc, national_modal) for oc in outcodes[~has_vote]],
             dtype=np.int32,
@@ -379,10 +474,82 @@ def _assign_home_force(
     return home
 
 
+def _window_annualised(
+    counts: np.ndarray,
+    months_in_year_force: np.ndarray,
+    home_fidx: np.ndarray,
+    usable: np.ndarray,
+    year_mask: np.ndarray,
+) -> np.ndarray:
+    """Raw annualised incidents/yr per (postcode, type) over a window of years.
+
+    For each force, the count is pooled over the force's covered months that fall
+    inside ``year_mask`` and annualised by those covered months. A covered
+    postcode with no incidents of a type gets 0; a postcode whose force never
+    published in the window, or whose geometry is unusable, gets NaN (unknown).
+    """
+    n_pc, n_types = counts.shape[0], counts.shape[1]
+    avg = np.full((n_pc, n_types), np.nan, dtype=np.float64)
+    for f in range(months_in_year_force.shape[0]):
+        sel = home_fidx == f
+        if not sel.any():
+            continue
+        cov_months = months_in_year_force[f].astype(np.float64) * year_mask
+        denom = cov_months.sum()
+        if denom <= 0:
+            continue  # force published nothing in this window; stays NaN
+        window_years = cov_months > 0
+        pooled = counts[sel][:, :, window_years].sum(axis=2, dtype=np.float64)
+        avg[sel] = pooled * 12.0 / denom
+    avg[~usable] = np.nan
+    return avg
+
+
+def _append_rollups(avg14: np.ndarray) -> np.ndarray:
+    """Append Serious/Minor rollup columns (sum of components) -> (n_pc, 16)."""
+    serious_idx = [ALL_CRIME_TYPES.index(t) for t in SERIOUS_CRIME_TYPES]
+    minor_idx = [ALL_CRIME_TYPES.index(t) for t in MINOR_CRIME_TYPES]
+    serious = avg14[:, serious_idx].sum(axis=1)
+    minor = avg14[:, minor_idx].sum(axis=1)
+    return np.column_stack([avg14, serious, minor])
+
+
+def _write_crime_table(
+    postcodes: np.ndarray,
+    raw_by_window: dict[str, np.ndarray],
+    output_path: Path,
+) -> None:
+    """Write the average-annual-count parquet (the filterable crime features).
+
+    ``raw_by_window[label]`` is the (n_postcodes, 16) average annual incident
+    count for that window (14 leaf types + 2 rollups). Each value is the raw,
+    absolute incidents/yr; a postcode with no usable data for a window keeps NaN
+    (written as null).
+    """
+    data: dict[str, np.ndarray] = {"postcode": postcodes}
+    for label, _years in WINDOWS:
+        counts = np.round(raw_by_window[label], 1).astype(np.float32)
+        for type_idx, name in enumerate(ALL_OUTPUT_TYPES):
+            data[crime_column(name, label)] = counts[:, type_idx]
+
+    _write_nan_aware(data, output_path, "postcode crime average annual counts")
+
+
+def _write_nan_aware(
+    data: dict[str, np.ndarray], output_path: Path, label: str
+) -> None:
+    frame = pl.DataFrame(data)
+    value_cols = [c for c in frame.columns if c != "postcode"]
+    frame = frame.with_columns(pl.col(c).fill_nan(None) for c in value_cols)
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    frame.write_parquet(output_path, compression="zstd")
+    print(f"Wrote {label}: {output_path}  {frame.shape}")
+
+
 def _rollup_long(
     long: pl.DataFrame, types: tuple[str, ...], rollup_name: str
 ) -> pl.DataFrame:
-    """Sum per-year annualised counts across ``types`` into a single rollup."""
+    """Sum per-year counts across ``types`` into a single rollup."""
     return (
         long.filter(pl.col("Crime type").is_in(list(types)))
         .group_by("postcode", "year")
@@ -392,108 +559,29 @@ def _rollup_long(
     )
 
 
-def _write_avg_yr(
-    postcodes: np.ndarray,
-    counts: np.ndarray,
-    months_in_year_force: np.ndarray,
-    home_fidx: np.ndarray,
-    norm: np.ndarray,
-    output_path: Path,
-) -> None:
-    """Write ``postcode`` + ``"{type} (avg/yr)"`` density-normalised averages.
-
-    The headline is the POOLED annualised rate over the home force's covered
-    months: ``sum(counts in covered years) * 12 / covered_months``. Years the
-    force published nothing contribute neither incidents nor months, so a
-    coverage gap (e.g. Greater Manchester 2019-07 onwards) is excluded instead
-    of read as zero crime. Pooling over the full covered window -- rather than
-    averaging only over years a type happened to occur -- is what keeps a
-    single robbery-year from printing as a perennial robbery rate. Each
-    postcode's value is then multiplied by ``norm`` (median_area / buffered
-    catchment area) so the metric is a density rather than a footprint-inflated
-    raw count; postcodes with unusable geometry (norm == 0) are null, not 0.
-    """
-    n_postcodes, n_types = counts.shape[0], counts.shape[1]
-    avg = np.full((n_postcodes, n_types), np.nan, dtype=np.float64)
-    for f in range(months_in_year_force.shape[0]):
-        sel = home_fidx == f
-        if not sel.any():
-            continue
-        cov_months = months_in_year_force[f].astype(np.float64)
-        denom = cov_months.sum()
-        if denom <= 0:
-            continue  # force never published; stays null
-        covered_years = cov_months > 0
-        pooled = counts[sel][:, :, covered_years].sum(axis=2, dtype=np.float64)
-        avg[sel] = pooled * 12.0 / denom
-
-    avg *= norm[:, None]
-    avg[norm <= 0] = np.nan  # unusable geometry: unknown, not zero
-    avg = np.round(avg, 1).astype(np.float32)
-
-    data: dict[str, np.ndarray] = {"postcode": postcodes}
-    for type_idx, name in enumerate(ALL_CRIME_TYPES):
-        data[f"{name} (avg/yr)"] = avg[:, type_idx]
-
-    # Serious/Minor rollup headlines = the exact SUM of their component (avg/yr)
-    # columns, so each rollup always equals the sum of the parts shown beside it
-    # and can never fall below one of its own components. All components share
-    # the postcode's pooled covered-month denominator, so the sum is itself the
-    # pooled rollup rate. Null components (unusable geometry) propagate to a
-    # null rollup.
-    for rollup_name, rollup_types in (
-        ("Serious crime", SERIOUS_CRIME_TYPES),
-        ("Minor crime", MINOR_CRIME_TYPES),
-    ):
-        rollup_idx = [ALL_CRIME_TYPES.index(name) for name in rollup_types]
-        data[f"{rollup_name} (avg/yr)"] = np.round(
-            avg[:, rollup_idx].sum(axis=1), 1
-        ).astype(np.float32)
-
-    frame = pl.DataFrame(data)
-    value_cols = [c for c in frame.columns if c != "postcode"]
-    frame = frame.with_columns(pl.col(c).fill_nan(None) for c in value_cols)
-
-    output_path.parent.mkdir(parents=True, exist_ok=True)
-    frame.write_parquet(output_path, compression="zstd")
-    print(f"Wrote postcode crime averages: {output_path}")
-
-
 def _write_by_year(
     postcodes: np.ndarray,
     counts: np.ndarray,
     years: list[int],
     months_in_year_force: np.ndarray,
     home_fidx: np.ndarray,
-    norm: np.ndarray,
+    usable: np.ndarray,
     min_bar_months: int,
     output_path: Path,
 ) -> None:
-    """Write nested ``"{type} (by year)"`` series plus rollups and coverage.
+    """Write nested ``"{type} (by year)"`` raw-count series plus rollups + coverage.
 
     A bar is only emitted for (postcode, year)s where the postcode's home force
-    published at least ``min_bar_months`` months -- annualising a thinner year
-    (x12 from a single month at the extreme) charts noise, and a force-gap year
-    must chart as *no data*, not zero. Bars are scaled by the force's covered
-    months in that year and area-normalised by the same ``norm`` factor as the
-    headline so chart and headline stay mutually consistent.
-
-    Every postcode gets a row (the output is dense) carrying ``covered_years``
-    -- the list of {year, months} the home force published at least
-    ``min_bar_months`` months -- so consumers can distinguish covered-but-
-    crime-free years (year listed, no bar => genuine zero) from coverage gaps
-    (year absent => unknown). Postcodes with unusable geometry get an empty
-    coverage list: their crime picture is unknown.
+    published at least ``min_bar_months`` months. Bars are the raw annualised
+    count for that year (``count * 12 / covered_months``); unlike the headline
+    windows there is no per-capita normalisation here -- the chart shows incident
+    volume over time. Every postcode gets a ``covered_years`` row so consumers can
+    distinguish covered-but-crime-free years from coverage gaps.
     """
-    # (n_postcodes, n_years): covered months of each postcode's home force.
     cov_pc_year = months_in_year_force[home_fidx, :]
-    usable = norm > 0
 
     annual = np.round(
-        counts.astype(np.float64)
-        * 12.0
-        / np.maximum(cov_pc_year[:, None, :], 1)
-        * norm[:, None, None],
+        counts.astype(np.float64) * 12.0 / np.maximum(cov_pc_year[:, None, :], 1),
         1,
     )
     bar_ok = (
@@ -506,9 +594,6 @@ def _write_by_year(
 
     type_names = np.array(ALL_CRIME_TYPES, dtype=object)
     year_values = np.array(years, dtype=np.int32)
-    # Explicit schema: with full masking (e.g. every year below min_bar_months)
-    # the fancy-indexed numpy object arrays are empty and polars would infer
-    # Object columns, which breaks the rollup `is_in` below.
     long = pl.DataFrame(
         {
             "postcode": postcodes[pc_i].astype(str),
@@ -532,8 +617,6 @@ def _write_by_year(
     type_cols = [c for c in wide.columns if c != "postcode"]
     wide = wide.rename({col: f"{col} (by year)" for col in type_cols})
 
-    # Dense base: every postcode, with its home force's coverage calendar.
-    # Built per force (there are ~45) and joined on the force index.
     coverage_per_force: list[list[dict[str, int]]] = []
     for f in range(months_in_year_force.shape[0]):
         coverage_per_force.append(
@@ -562,7 +645,6 @@ def _write_by_year(
     dense = (
         base.join(coverage_frame, on="_fidx", how="left")
         .with_columns(
-            # Unusable geometry: empty coverage -- the crime picture is unknown.
             pl.when(pl.col("_usable"))
             .then(pl.col(COVERAGE_COLUMN))
             .otherwise(pl.col(COVERAGE_COLUMN).list.head(0))
@@ -577,11 +659,47 @@ def _write_by_year(
     print(f"Wrote postcode crime by-year series: {output_path}  {wide.shape}")
 
 
+def _finalize_records(
+    records_shard_dir: Path, output_path: Path
+) -> None:
+    """Concatenate the per-batch record shards into one postcode-sorted parquet.
+
+    Sorting by postcode lets the server build a contiguous per-postcode slice
+    index. The sort runs on the streaming engine so it spills rather than holding
+    all ~40M rows in memory.
+    """
+    shards = sorted(records_shard_dir.glob("*.parquet"))
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    if not shards:
+        pl.DataFrame(
+            schema={
+                "postcode": pl.String,
+                "month_index": pl.Int32,
+                "crime_type": pl.String,
+                "location": pl.String,
+                "outcome": pl.String,
+                "lat": pl.Float32,
+                "lon": pl.Float32,
+            }
+        ).write_parquet(output_path, compression="zstd")
+        print(f"Wrote crime records: {output_path}  (empty)")
+        return
+
+    (
+        pl.scan_parquet(shards)
+        .sort("postcode")
+        .sink_parquet(output_path, compression="zstd")
+    )
+    n = pl.scan_parquet(output_path).select(pl.len()).collect().item()
+    print(f"Wrote crime records: {output_path}  ({n:,} rows)")
+
+
 def transform_crime_spatial(
     crime_dir: Path,
     boundaries_dir: Path,
     output_path: Path,
     by_year_output_path: Path,
+    records_output_path: Path,
     buffer_m: float = DEFAULT_BUFFER_M,
     max_postcodes: int | None = None,
     max_files: int | None = None,
@@ -594,6 +712,10 @@ def transform_crime_spatial(
         csvs = csvs[:max_files]
 
     years, forces, months_in_year_force = _force_calendar(csvs)
+    latest_year = years[-1]
+    # Records cover the longest window's calendar years (January of its earliest
+    # year onward), so they reconcile exactly with the calendar-year headline.
+    records_min_ym = (latest_year - (RECORDS_WINDOW_YEARS - 1)) * 12
     print(
         f"Found {len(csvs):,} street crime CSVs across {len(forces)} forces "
         f"({years[0]}-{years[-1]})"
@@ -601,27 +723,10 @@ def transform_crime_spatial(
     )
 
     postcodes, polygons = load_postcode_polygons(boundaries_dir, max_postcodes)
+    postcodes = np.asarray(postcodes)
 
     print(f"Buffering {len(postcodes):,} postcode polygons by {buffer_m:g}m...")
-    buffers, tree = _build_tree(polygons, buffer_m)
-
-    # Area-normalisation factor (median_area / catchment_area): divides out the
-    # size of each postcode's catchment so the count measures crime density, not
-    # how much ground the buffer sweeps. We normalise by the *buffered* area --
-    # the region that actually collects points -- rather than the raw polygon, so
-    # a tiny unit postcode isn't over-inflated by the fixed buffer-ring floor.
-    # Buffers are in EPSG:27700, so shapely.area is in m^2.
-    areas = shapely.area(buffers).astype(np.float64)
-    usable_area = np.isfinite(areas) & (areas > 0)
-    if not usable_area.any():
-        raise ValueError("No postcode buffers have a positive area to normalise by")
-    median_area = float(np.median(areas[usable_area]))
-    norm = np.zeros(len(postcodes), dtype=np.float64)
-    norm[usable_area] = median_area / areas[usable_area]
-    print(
-        f"Area-normalising to median catchment area {median_area:,.0f} m^2 "
-        f"({int(usable_area.sum()):,}/{len(areas):,} postcodes have usable area)"
-    )
+    tree, usable = _build_tree(polygons, buffer_m)
 
     type_to_idx = {name: idx for idx, name in enumerate(ALL_CRIME_TYPES)}
     year_to_idx = {year: idx for idx, year in enumerate(years)}
@@ -630,25 +735,50 @@ def transform_crime_spatial(
     force_votes = np.zeros((len(postcodes), len(forces)), dtype=np.int32)
 
     transformer = Transformer.from_crs("EPSG:4326", "EPSG:27700", always_xy=True)
-    _accumulate_counts(
-        csvs, tree, type_to_idx, year_to_idx, force_to_idx, transformer, counts, force_votes
-    )
 
-    home_fidx = _assign_home_force(np.asarray(postcodes), force_votes, forces)
+    with tempfile.TemporaryDirectory(
+        prefix="crime_records_", dir=records_output_path.parent
+    ) as shard_dir_str:
+        shard_dir = Path(shard_dir_str)
+        _accumulate_counts(
+            csvs,
+            postcodes,
+            tree,
+            type_to_idx,
+            year_to_idx,
+            force_to_idx,
+            transformer,
+            counts,
+            force_votes,
+            shard_dir,
+            records_min_ym,
+        )
 
-    _write_avg_yr(
-        postcodes, counts, months_in_year_force, home_fidx, norm, output_path
-    )
-    _write_by_year(
-        postcodes,
-        counts,
-        years,
-        months_in_year_force,
-        home_fidx,
-        norm,
-        min_bar_months,
-        by_year_output_path,
-    )
+        home_fidx = _assign_home_force(postcodes, force_votes, forces)
+
+        # Per-window raw annualised averages (14 leaf types + Serious/Minor).
+        raw_by_window: dict[str, np.ndarray] = {}
+        for label, win_years in WINDOWS:
+            year_mask = np.array(
+                [1.0 if y > latest_year - win_years else 0.0 for y in years]
+            )
+            avg14 = _window_annualised(
+                counts, months_in_year_force, home_fidx, usable, year_mask
+            )
+            raw_by_window[label] = _append_rollups(avg14)
+
+        _write_crime_table(postcodes, raw_by_window, output_path)
+        _write_by_year(
+            postcodes,
+            counts,
+            years,
+            months_in_year_force,
+            home_fidx,
+            usable,
+            min_bar_months,
+            by_year_output_path,
+        )
+        _finalize_records(shard_dir, records_output_path)
 
 
 def main() -> None:
@@ -671,7 +801,7 @@ def main() -> None:
         "--output",
         type=Path,
         required=True,
-        help="Output parquet: postcode + '{type} (avg/yr)' columns",
+        help="Output parquet: postcode + '{type} (/yr, <window>)' average-annual-count columns",
     )
     parser.add_argument(
         "--output-by-year",
@@ -680,10 +810,10 @@ def main() -> None:
         help="Output parquet: postcode + nested '{type} (by year)' columns",
     )
     parser.add_argument(
-        "--buffer-m",
-        type=float,
-        default=DEFAULT_BUFFER_M,
-        help="Outward buffer (metres) added to each postcode boundary",
+        "--output-records",
+        type=Path,
+        required=True,
+        help="Output parquet: one row per counted incident (last 7 years), postcode-sorted",
     )
     parser.add_argument(
         "--max-postcodes",
@@ -705,15 +835,12 @@ def main() -> None:
     )
     args = parser.parse_args()
 
-    if args.buffer_m <= 0:
-        raise SystemExit("--buffer-m must be greater than zero")
-
     transform_crime_spatial(
         crime_dir=args.input,
         boundaries_dir=args.boundaries,
         output_path=args.output,
         by_year_output_path=args.output_by_year,
-        buffer_m=args.buffer_m,
+        records_output_path=args.output_records,
         max_postcodes=args.max_postcodes,
         max_files=args.max_files,
         min_bar_months=args.min_bar_months,