lgtm
This commit is contained in:
parent
982e0cc89c
commit
cfaf58dfba
44 changed files with 793 additions and 201 deletions
|
|
@ -1,43 +1,43 @@
|
|||
# postcode_boundaries
|
||||
|
||||
Synthesizes postcode boundary polygons for England and Wales from three datasets. UK postcodes don't have official boundary polygons — Royal Mail defines postcodes as sets of delivery addresses, not geographic areas. This pipeline constructs a plausible polygon for every postcode by combining Output Area boundaries, UPRN point locations, and INSPIRE cadastral parcels.
|
||||
Synthesizes postcode boundary polygons for England and Wales from three datasets. UK postcodes don't have official boundary polygons: Royal Mail defines postcodes as sets of delivery addresses, not geographic areas. This pipeline constructs a plausible polygon for every postcode by combining Output Area boundaries, UPRN point locations, and INSPIRE cadastral parcels.
|
||||
|
||||
## The three input datasets
|
||||
|
||||
**1. Output Area (OA) boundaries** — ONS Census Output Areas are the smallest geographic unit in the UK census (~125 households each). They tile all of England and Wales with no gaps or overlaps. Stored in a GeoPackage in British National Grid (EPSG:27700, meters). ~190,000 OAs.
|
||||
**1. Output Area (OA) boundaries**: ONS Census Output Areas are the smallest geographic unit in the UK census (~125 households each). They tile all of England and Wales with no gaps or overlaps. Stored in a GeoPackage in British National Grid (EPSG:27700, meters). ~190,000 OAs.
|
||||
|
||||
**2. UPRN lookup** — Every Unique Property Reference Number in England and Wales, with its grid coordinates (easting/northing in BNG), its postcode (`PCDS`), and its OA code (`OA21CD`). ~37 million rows. This is the critical bridge: it tells you which postcodes exist inside each OA, and where each address physically sits.
|
||||
**2. UPRN lookup**: Every Unique Property Reference Number in England and Wales, with its grid coordinates (easting/northing in BNG), its postcode (`PCDS`), and its OA code (`OA21CD`). ~37 million rows. This is the critical bridge: it tells you which postcodes exist inside each OA, and where each address physically sits.
|
||||
|
||||
**3. INSPIRE Index Polygons** — Land Registry cadastral parcels covering most of England and Wales. Each ZIP contains a GML file with polygon coordinate lists representing individual land parcels (buildings, plots of land). ~24 million polygons. These give fine-grained building/plot outlines that are much more precise than anything you could derive from point locations alone.
|
||||
**3. INSPIRE Index Polygons**: Land Registry cadastral parcels covering most of England and Wales. Each ZIP contains a GML file with polygon coordinate lists representing individual land parcels (buildings, plots of land). ~24 million polygons. These give fine-grained building/plot outlines that are much more precise than anything you could derive from point locations alone.
|
||||
|
||||
## The four phases
|
||||
|
||||
### Phase 1: Loading data
|
||||
|
||||
**OA boundaries** (`oa_boundaries.py`): Opens the GeoPackage via SQLite, reads every row from `OA_2021_EW_BGC_V2`. Each row's `SHAPE` column is a GeoPackage binary blob — a standard 8-byte header, then a variable-size envelope (bounding box), then WKB geometry. `parse_gpkg_geometry` reads byte 3 to extract the envelope type (0-4), looks up the envelope size, skips past the header, and hands the remaining WKB bytes to Shapely. Single-polygon MultiPolygons are unwrapped. Result: `dict[oa_code, Polygon]`, all in BNG.
|
||||
**OA boundaries** (`oa_boundaries.py`): Opens the GeoPackage via SQLite, reads every row from `OA_2021_EW_BGC_V2`. Each row's `SHAPE` column is a GeoPackage binary blob: a standard 8-byte header, then a variable-size envelope (bounding box), then WKB geometry. `parse_gpkg_geometry` reads byte 3 to extract the envelope type (0-4), looks up the envelope size, skips past the header, and hands the remaining WKB bytes to Shapely. Single-polygon MultiPolygons are unwrapped. Result: `dict[oa_code, Polygon]`, all in BNG.
|
||||
|
||||
**UPRNs** (`uprn.py`): The raw parquet has far more columns than needed. The lazy scan selects only four columns, filters out Scotland (OA codes starting with `S`), drops nulls and blank postcodes (stripping whitespace first), then sorts by OA code. The sort uses `sink_parquet` to write to a temp file — this avoids polars doubling memory from an in-memory sort on ~37M rows.
|
||||
**UPRNs** (`uprn.py`): The raw parquet has far more columns than needed. The lazy scan selects only four columns, filters out Scotland (OA codes starting with `S`), drops nulls and blank postcodes (stripping whitespace first), then sorts by OA code. The sort uses `sink_parquet` to write to a temp file. This avoids polars doubling memory from an in-memory sort on ~37M rows.
|
||||
|
||||
After reading the sorted file back, it builds an offset dictionary. Rather than grouping into Python lists (which would create 37M Python string objects), it detects group boundaries by comparing each row's OA code to the previous row's. The result is `offsets[oa_code] = (start_row, end_row)` — a simple slice into the DataFrame. The OA column is then dropped since it's no longer needed, saving ~400MB.
|
||||
After reading the sorted file back, it builds an offset dictionary. Rather than grouping into Python lists (which would create 37M Python string objects), it detects group boundaries by comparing each row's OA code to the previous row's. The result is `offsets[oa_code] = (start_row, end_row)`: a simple slice into the DataFrame. The OA column is then dropped since it's no longer needed, saving ~400MB.
|
||||
|
||||
`get_oa_uprns` later retrieves a single OA's data by slicing `df[start:end]` and extracting the coordinates and postcodes.
|
||||
|
||||
### Phase 2: INSPIRE data
|
||||
|
||||
INSPIRE comes as ~350 ZIP files, each containing a GML file with thousands of `PREDEFINED` elements. Each element has a `posList` — a flat string of coordinate pairs.
|
||||
INSPIRE comes as ~350 ZIP files, each containing a GML file with thousands of `PREDEFINED` elements. Each element has a `posList`: a flat string of coordinate pairs.
|
||||
|
||||
**Parsing** (`inspire.py:parse_inspire_zip`): Uses `iterparse` for streaming XML parsing (constant memory per ZIP). For each `PREDEFINED` element, extracts the `posList` text, splits into floats, reshapes to Nx2. Calls `elem.clear()` after each element to free XML nodes immediately.
|
||||
|
||||
**Caching** (`inspire.py:cache_inspire`): Parsing 350 ZIPs takes a while, so results are cached as three files:
|
||||
- `inspire_coords.bin` — flat binary dump of all float64 coordinate pairs, streamed to disk as each ZIP is parsed
|
||||
- `inspire_bboxes.npy` — (N, 4) array of `[min_e, min_n, max_e, max_n]` per polygon
|
||||
- `inspire_offsets.npy` — (N, 2) array of `[byte_offset_into_coords_bin, n_points]`
|
||||
- `inspire_coords.bin`: flat binary dump of all float64 coordinate pairs, streamed to disk as each ZIP is parsed
|
||||
- `inspire_bboxes.npy`: (N, 4) array of `[min_e, min_n, max_e, max_n]` per polygon
|
||||
- `inspire_offsets.npy`: (N, 2) array of `[byte_offset_into_coords_bin, n_points]`
|
||||
|
||||
Pre-allocates numpy arrays at 25M capacity and grows by 1.5x if needed (using in-place `resize` with `refcheck=False`). This avoids Python list overhead for 24M polygons. The coords file is written sequentially — each polygon's raw bytes are appended, and its byte offset is recorded.
|
||||
Pre-allocates numpy arrays at 25M capacity and grows by 1.5x if needed (using in-place `resize` with `refcheck=False`). This avoids Python list overhead for 24M polygons. The coords file is written sequentially: each polygon's raw bytes are appended, and its byte offset is recorded.
|
||||
|
||||
**Loading** (`inspire.py:load_inspire`): Bboxes and offsets are loaded into RAM (~1.1GB). Coords are memory-mapped — the OS pages them in on demand from the ~3GB file, never loading the whole thing.
|
||||
**Loading** (`inspire.py:load_inspire`): Bboxes and offsets are loaded into RAM (~1.1GB). Coords are memory-mapped: the OS pages them in on demand from the ~3GB file, never loading the whole thing.
|
||||
|
||||
**Candidate retrieval** (`inspire.py:InspireIndex`): A uniform 1km grid index is built once over the 24M parcel bboxes (`build_inspire_index`). Each OA lookup (`InspireIndex.candidates`) gathers parcels from the cells its bounding box covers plus a small overflow list of parcels larger than one cell, then applies the exact bbox overlap test — O(cells + candidates) instead of an O(24M) scan per OA (the old linear scan was ~4h of the run on its own). The candidate set and order are identical to the scan. Typically matches 10-500 parcels per OA, materialized as Shapely Polygon objects by reading their coordinate slice from the memory-mapped file; invalid polygons are repaired with `make_valid`.
|
||||
**Candidate retrieval** (`inspire.py:InspireIndex`): A uniform 1km grid index is built once over the 24M parcel bboxes (`build_inspire_index`). Each OA lookup (`InspireIndex.candidates`) gathers parcels from the cells its bounding box covers plus a small overflow list of parcels larger than one cell, then applies the exact bbox overlap test: O(cells + candidates) instead of an O(24M) scan per OA (the old linear scan was ~4h of the run on its own). The candidate set and order are identical to the scan. Typically matches 10-500 parcels per OA, materialized as Shapely Polygon objects by reading their coordinate slice from the memory-mapped file; invalid polygons are repaired with `make_valid`.
|
||||
|
||||
### Phase 3: Processing OAs
|
||||
|
||||
|
|
@ -49,7 +49,7 @@ The main loop in `__main__.py` (`_process_oas`) iterates through every OA that h
|
|||
|
||||
#### Stage A: INSPIRE-based claiming
|
||||
|
||||
Build an STRtree spatial index over the INSPIRE candidate polygons. Convert all UPRN points to Shapely Point objects and batch-query the tree with `predicate="intersects"`. This returns pairs of (point_index, candidate_index) — which UPRNs fall inside which parcels.
|
||||
Build an STRtree spatial index over the INSPIRE candidate polygons. Convert all UPRN points to Shapely Point objects and batch-query the tree with `predicate="intersects"`. This returns pairs of (point_index, candidate_index): which UPRNs fall inside which parcels.
|
||||
|
||||
For each INSPIRE parcel that contains at least one UPRN, run a majority vote: whichever postcode has the most UPRNs inside that parcel wins the parcel. Accumulate winning parcels per postcode, union them, and clip to the OA boundary. The result is `claimed[postcode] = polygon_within_oa`.
|
||||
|
||||
|
|
@ -73,15 +73,15 @@ The effect: every non-parcel patch of OA gets assigned to the nearest postcode b
|
|||
|
||||
Each postcode gets its INSPIRE-claimed polygon (if any) plus its Voronoi share (if any). These are unioned together, validated, and stripped of any non-polygonal geometry debris from `make_valid`.
|
||||
|
||||
The output of `process_oa` is `list[(postcode, polygon)]` — the per-OA fragments. A single postcode that spans two OAs produces two separate fragments (one from each OA's processing).
|
||||
The output of `process_oa` is `list[(postcode, polygon)]`: the per-OA fragments. A single postcode that spans two OAs produces two separate fragments (one from each OA's processing).
|
||||
|
||||
### Phase 4: Merging and writing
|
||||
|
||||
**Fragment merging** (`output.py:merge_fragments`): Groups all fragments by postcode, unions them. If the result is a MultiPolygon (meaning the postcode has disconnected pieces — either from spanning OAs with a gap, or algorithm artifacts), applies a 5m buffer-then-unbuffer to close tiny gaps from floating-point mismatches at OA boundary edges. If still a MultiPolygon after that, keeps the largest part **plus any other part ≥ `_MIN_DETACHED_PART_AREA` (100 m²)** (`_keep_polygon_parts`); only sub-100 m² noise slivers are dropped. Keeping substantial detached parts matters because a postcode genuinely split across an OA seam (by a railway, river, or main road wider than the 5m buffer) would otherwise lose a chunk — measured at ~1.8% of merged area left as uncovered gaps (often 3000–5000 m² building blocks) before this change.
|
||||
**Fragment merging** (`output.py:merge_fragments`): Groups all fragments by postcode, unions them. If the result is a MultiPolygon (meaning the postcode has disconnected pieces: either from spanning OAs with a gap, or algorithm artifacts), applies a 5m buffer-then-unbuffer to close tiny gaps from floating-point mismatches at OA boundary edges. If still a MultiPolygon after that, keeps the largest part **plus any other part ≥ `_MIN_DETACHED_PART_AREA` (100 m²)** (`_keep_polygon_parts`); only sub-100 m² noise slivers are dropped. Keeping substantial detached parts matters because a postcode genuinely split across an OA seam (by a railway, river, or main road wider than the 5m buffer) would otherwise lose a chunk, measured at ~1.8% of merged area left as uncovered gaps (often 3000–5000 m² building blocks) before this change.
|
||||
|
||||
**Greenspace subtraction is connectivity-preserving** (`greenspace.py:subtract_greenspace`): park/water polygons are subtracted from each postcode, but greenspace that *crosses* a postcode (a river, a strip of parkland, a golf course through a village) would otherwise split it into scattered pieces. When the subtraction disconnects a postcode, `_reconnect_split` re-adds the narrowest removed necks — a morphological closing (`_RECONNECT_BRIDGE_M`, 25 m) clipped to the original postcode footprint — so parts ≤ ~50 m apart stay joined by a thin bridge of the postcode's own land (no address moves); genuinely wide barriers stay subtracted and the postcode legitimately splits.
|
||||
**Greenspace subtraction is connectivity-preserving** (`greenspace.py:subtract_greenspace`): park/water polygons are subtracted from each postcode, but greenspace that *crosses* a postcode (a river, a strip of parkland, a golf course through a village) would otherwise split it into scattered pieces. When the subtraction disconnects a postcode, `_reconnect_split` re-adds the narrowest removed necks, a morphological closing (`_RECONNECT_BRIDGE_M`, 25 m) clipped to the original postcode footprint, so parts ≤ ~50 m apart stay joined by a thin bridge of the postcode's own land (no address moves); genuinely wide barriers stay subtracted and the postcode legitimately splits.
|
||||
|
||||
**GeoJSON output** (`output.py:write_district_geojson`): three passes. Pass 1 converts every postcode from BNG to WGS84 (pyproj), simplifies with 1m tolerance (Douglas-Peucker), and snaps to 6 decimal places (~0.1m precision); multi-part postcodes become `MultiPolygon` (`to_wgs84_geojson_multi`, each part handled independently), single-part stay `Polygon`. The whole set is then made a **partition** (`_resolve_overlaps`): each postcode is trimmed by the union of its higher-priority overlapping neighbours, where **priority = ascending area** (smaller postcodes win contested ground). That single rule handles both seam overlap *and* containment — an enclosed postcode is always smaller than its container, so it keeps its area while the container gets a hole (the query uses both the `overlaps` and `contains` predicates, since `overlaps` alone excludes containment). This runs last, so nothing re-introduces overlap; a postcode that would be emptied keeps its original geometry, so no active postcode is dropped. Pass 2 **de-fragments** the partition (`_eliminate_small_detached_parts`): a detached part that is *both* small in absolute terms (< `_ELIM_ABS_MAX_M2`, 3000 m²) *and* a minor fraction (< `_ELIM_FRAC_MAX`, 15%) of its postcode is absorbed into the neighbouring postcode it shares the most boundary with — the classic GIS *eliminate*. This removes the Voronoi/INSPIRE/seam *scatter* that left ~1/3 of postcodes non-contiguous, while a genuine bisection (two substantial parts split by a river/railway) keeps both parts. The land is **reassigned**, never dropped, so the output stays a gapless partition and coverage is conserved; the largest part of every postcode is always retained, so no active postcode is dropped (a tiny neighbour-less sliver in removed greenspace is dropped, a larger isolated patch is kept). Pass 3 groups postcodes by district (the outward code, e.g. `SW1A` from `SW1A 1AA`), rounds coordinates to 6dp, and writes a `{district}.geojson` FeatureCollection. Each Feature has `postcodes` (formatted like `"SW1A 1AA"`) and `mapit_code` (no space: `"SW1A1AA"`) in its properties.
|
||||
**GeoJSON output** (`output.py:write_district_geojson`): three passes. Pass 1 converts every postcode from BNG to WGS84 (pyproj), simplifies with 1m tolerance (Douglas-Peucker), and snaps to 6 decimal places (~0.1m precision); multi-part postcodes become `MultiPolygon` (`to_wgs84_geojson_multi`, each part handled independently), single-part stay `Polygon`. The whole set is then made a **partition** (`_resolve_overlaps`): each postcode is trimmed by the union of its higher-priority overlapping neighbours, where **priority = ascending area** (smaller postcodes win contested ground). That single rule handles both seam overlap *and* containment: an enclosed postcode is always smaller than its container, so it keeps its area while the container gets a hole (the query uses both the `overlaps` and `contains` predicates, since `overlaps` alone excludes containment). This runs last, so nothing re-introduces overlap; a postcode that would be emptied keeps its original geometry, so no active postcode is dropped. Pass 2 **de-fragments** the partition (`_eliminate_small_detached_parts`): a detached part that is *both* small in absolute terms (< `_ELIM_ABS_MAX_M2`, 3000 m²) *and* a minor fraction (< `_ELIM_FRAC_MAX`, 15%) of its postcode is absorbed into the neighbouring postcode it shares the most boundary with: the classic GIS *eliminate*. This removes the Voronoi/INSPIRE/seam *scatter* that left ~1/3 of postcodes non-contiguous, while a genuine bisection (two substantial parts split by a river/railway) keeps both parts. The land is **reassigned**, never dropped, so the output stays a gapless partition and coverage is conserved; the largest part of every postcode is always retained, so no active postcode is dropped (a tiny neighbour-less sliver in removed greenspace is dropped, a larger isolated patch is kept). Pass 3 groups postcodes by district (the outward code, e.g. `SW1A` from `SW1A 1AA`), rounds coordinates to 6dp, and writes a `{district}.geojson` FeatureCollection. Each Feature has `postcodes` (formatted like `"SW1A 1AA"`) and `mapit_code` (no space: `"SW1A1AA"`) in its properties.
|
||||
|
||||
## Memory architecture
|
||||
|
||||
|
|
@ -97,7 +97,7 @@ The pipeline is designed to run in <12GB:
|
|||
| Fragments | Python list of (str, Shapely) | grows during processing |
|
||||
|
||||
Key design choices:
|
||||
- INSPIRE coords are memory-mapped, not loaded — the OS pages in only the ~100-500 polygons needed per OA
|
||||
- INSPIRE coords are memory-mapped, not loaded: the OS pages in only the ~100-500 polygons needed per OA
|
||||
- UPRNs sorted + offset dict avoids per-OA groupby allocation
|
||||
- `sink_parquet` for the sort avoids doubling memory
|
||||
- `release_memory()` calls `gc.collect()` + glibc `malloc_trim(0)` to return freed pages to the OS between phases
|
||||
|
|
@ -105,25 +105,25 @@ Key design choices:
|
|||
|
||||
## Key invariants
|
||||
|
||||
1. **No two postcodes cover the same ground in the output** — within an OA the INSPIRE claiming + Voronoi tile it with no overlap, and a final `_resolve_overlaps` partition pass removes the thin overlap strips that the merge buffer + per-postcode simplification introduce across OA seams (measured residual overlap ~0.01% of area)
|
||||
2. **Every postcode that exists in the UPRN data gets a polygon** — unless all its UPRNs share coordinates with another postcode's UPRNs (handled by jitter) or it has zero UPRNs
|
||||
3. **Postcode polygons never extend outside their OA(s)** — all geometry is clipped to OA boundaries
|
||||
4. **A postcode split across an OA seam keeps all its substantial parts** — `merge_fragments` keeps every part ≥ 100 m² and the output is emitted as a `MultiPolygon` (the Rust server `postcodes.rs` and `loader.py` both parse MultiPolygon); only sub-100 m² noise slivers are dropped
|
||||
5. **Postcodes are contiguous unless genuinely split** — most non-contiguity is *scatter* (a unit drawn as many disconnected specks) from point-Voronoi over sparse/interleaved UPRNs, greenspace cutting across a unit, and overlap/seam slivers. Connectivity-preserving greenspace subtraction + the `_eliminate_small_detached_parts` de-fragmentation pass absorb that scatter into neighbours (coverage-conserving), cutting the share of multi-part postcodes roughly in half (~30% → ~14% measured on the worst rural/coastal districts) without dropping any postcode or leaving coverage gaps. Genuine bisections (river/railway/major road, or a detached part above the absolute+fraction thresholds) are preserved.
|
||||
1. **No two postcodes cover the same ground in the output**: within an OA the INSPIRE claiming + Voronoi tile it with no overlap, and a final `_resolve_overlaps` partition pass removes the thin overlap strips that the merge buffer + per-postcode simplification introduce across OA seams (measured residual overlap ~0.01% of area)
|
||||
2. **Every postcode that exists in the UPRN data gets a polygon**: unless all its UPRNs share coordinates with another postcode's UPRNs (handled by jitter) or it has zero UPRNs
|
||||
3. **Postcode polygons never extend outside their OA(s)**: all geometry is clipped to OA boundaries
|
||||
4. **A postcode split across an OA seam keeps all its substantial parts**: `merge_fragments` keeps every part ≥ 100 m² and the output is emitted as a `MultiPolygon` (the Rust server `postcodes.rs` and `loader.py` both parse MultiPolygon); only sub-100 m² noise slivers are dropped
|
||||
5. **Postcodes are contiguous unless genuinely split**: most non-contiguity is *scatter* (a unit drawn as many disconnected specks) from point-Voronoi over sparse/interleaved UPRNs, greenspace cutting across a unit, and overlap/seam slivers. Connectivity-preserving greenspace subtraction + the `_eliminate_small_detached_parts` de-fragmentation pass absorb that scatter into neighbours (coverage-conserving), cutting the share of multi-part postcodes roughly in half (~30% → ~14% measured on the worst rural/coastal districts) without dropping any postcode or leaving coverage gaps. Genuine bisections (river/railway/major road, or a detached part above the absolute+fraction thresholds) are preserved.
|
||||
|
||||
## Module structure
|
||||
|
||||
```
|
||||
postcode_boundaries/
|
||||
__init__.py — Package docstring
|
||||
__main__.py — CLI entry point, four-phase orchestration
|
||||
memory.py — release_memory() glibc malloc_trim helper
|
||||
oa_boundaries.py — GeoPackage parsing, OA boundary loading
|
||||
uprn.py — UPRN loading (sorted DataFrame + offset dict), per-OA access
|
||||
inspire.py — INSPIRE GML parsing, caching, loading, bbox candidate retrieval
|
||||
voronoi.py — Voronoi region computation clipped to boundary
|
||||
process_oa.py — Per-OA processing (INSPIRE assignment + Voronoi fallback)
|
||||
output.py — BNG to WGS84 transform, fragment merging, GeoJSON writing
|
||||
__init__.py : Package docstring
|
||||
__main__.py : CLI entry point, four-phase orchestration
|
||||
memory.py : release_memory() glibc malloc_trim helper
|
||||
oa_boundaries.py : GeoPackage parsing, OA boundary loading
|
||||
uprn.py : UPRN loading (sorted DataFrame + offset dict), per-OA access
|
||||
inspire.py : INSPIRE GML parsing, caching, loading, bbox candidate retrieval
|
||||
voronoi.py : Voronoi region computation clipped to boundary
|
||||
process_oa.py : Per-OA processing (INSPIRE assignment + Voronoi fallback)
|
||||
output.py : BNG to WGS84 transform, fragment merging, GeoJSON writing
|
||||
```
|
||||
|
||||
Invoked as:
|
||||
|
|
|
|||
|
|
@ -192,7 +192,7 @@ def build_fragments(args: argparse.Namespace) -> list[Fragment]:
|
|||
print("Phase 3: Processing OAs")
|
||||
print("=" * 60)
|
||||
|
||||
# Build work list — precompute which OAs are single vs multi-postcode
|
||||
# Build work list: precompute which OAs are single vs multi-postcode
|
||||
oa_codes_with_data = sorted(set(oa_geoms.keys()) & set(uprn_offsets.keys()))
|
||||
skipped_no_uprn = len(oa_geoms) - len(oa_codes_with_data)
|
||||
skipped_no_boundary = len(uprn_offsets) - len(oa_codes_with_data)
|
||||
|
|
@ -275,7 +275,7 @@ def main() -> None:
|
|||
|
||||
if use_cache and fragments_cache_is_fresh(fragments_cache, fragment_inputs):
|
||||
print("=" * 60)
|
||||
print("Phase 3 cache hit — loading fragments (skipping Phases 1-3)")
|
||||
print("Phase 3 cache hit: loading fragments (skipping Phases 1-3)")
|
||||
print("=" * 60)
|
||||
all_fragments = load_fragments(fragments_cache)
|
||||
print(
|
||||
|
|
|
|||
|
|
@ -81,7 +81,7 @@ class TestFirstOADropped:
|
|||
"""E00000001 is the first OA after sorting. It must appear in offsets."""
|
||||
df, offsets = load_uprns(uprn_parquet)
|
||||
assert "E00000001" in offsets, (
|
||||
"First OA (E00000001) missing from offsets — shift(1) null comparison bug"
|
||||
"First OA (E00000001) missing from offsets: shift(1) null comparison bug"
|
||||
)
|
||||
|
||||
def test_all_oas_present(self, uprn_parquet):
|
||||
|
|
@ -221,7 +221,7 @@ class TestWhitespacePostcodes:
|
|||
|
||||
# The remapped point must be grouped under the successor's OA, not the
|
||||
# terminated postcode's OA.
|
||||
assert "E00000002" in offsets, "Successor OA missing — remap kept old OA"
|
||||
assert "E00000002" in offsets, "Successor OA missing: remap kept old OA"
|
||||
assert "E00000001" not in offsets, (
|
||||
"Remapped point still lives in the terminated postcode's OA"
|
||||
)
|
||||
|
|
@ -304,10 +304,10 @@ class TestVoronoiDeduplication:
|
|||
# Plus postcode A has one at a different location
|
||||
points = np.array(
|
||||
[
|
||||
[500020, 180050], # postcode A — unique location
|
||||
[500050, 180050], # postcode A — shared location
|
||||
[500050, 180050], # postcode B — shared location (same coords)
|
||||
[500080, 180050], # postcode B — unique location
|
||||
[500020, 180050], # postcode A: unique location
|
||||
[500050, 180050], # postcode A: shared location
|
||||
[500050, 180050], # postcode B: shared location (same coords)
|
||||
[500080, 180050], # postcode B: unique location
|
||||
]
|
||||
)
|
||||
postcodes = ["A", "A", "B", "B"]
|
||||
|
|
@ -321,7 +321,7 @@ class TestVoronoiDeduplication:
|
|||
points = np.array(
|
||||
[
|
||||
[500050, 180050], # postcode A
|
||||
[500050, 180050], # postcode B — identical coords
|
||||
[500050, 180050], # postcode B: identical coords
|
||||
]
|
||||
)
|
||||
postcodes = ["A", "B"]
|
||||
|
|
@ -362,11 +362,11 @@ class TestVoronoiCoincidentClusterNotCrushed:
|
|||
boundary = box(0, 0, 1000, 1000)
|
||||
points = np.array(
|
||||
[
|
||||
[500, 500], # A — coincident
|
||||
[500, 500], # B — coincident
|
||||
[500, 500], # C — coincident
|
||||
[500, 500], # D — coincident
|
||||
[100, 100], # OUT — elsewhere in the OA
|
||||
[500, 500], # A: coincident
|
||||
[500, 500], # B: coincident
|
||||
[500, 500], # C: coincident
|
||||
[500, 500], # D: coincident
|
||||
[100, 100], # OUT: elsewhere in the OA
|
||||
],
|
||||
dtype=np.float64,
|
||||
)
|
||||
|
|
@ -412,7 +412,7 @@ class TestVoronoiCollinear:
|
|||
"""Collinear points (handled by dummy corners) must distribute area fairly."""
|
||||
|
||||
def test_collinear_points_all_postcodes_get_area(self, square_boundary):
|
||||
"""Points along a line — every postcode must get area."""
|
||||
"""Points along a line: every postcode must get area."""
|
||||
points = np.array(
|
||||
[
|
||||
[500020, 180050],
|
||||
|
|
@ -483,14 +483,14 @@ class TestProcessOAGeometryTypes:
|
|||
def test_overlapping_inspire_no_postcode_overlap(self):
|
||||
"""Overlapping INSPIRE parcels assigned to different postcodes must not overlap."""
|
||||
oa_geom = box(500000, 180000, 500100, 180100)
|
||||
# Two overlapping parcels — left half and a wider middle section
|
||||
# Two overlapping parcels: left half and a wider middle section
|
||||
parcel_left = box(500000, 180000, 500060, 180100)
|
||||
parcel_right = box(500040, 180000, 500100, 180100) # overlaps left by 20m
|
||||
# UPRN in left parcel → postcode A, UPRN in right parcel → postcode B
|
||||
points = np.array(
|
||||
[
|
||||
[500020, 180050], # postcode A — inside left parcel
|
||||
[500080, 180050], # postcode B — inside right parcel
|
||||
[500020, 180050], # postcode A: inside left parcel
|
||||
[500080, 180050], # postcode B: inside right parcel
|
||||
]
|
||||
)
|
||||
postcodes = ["A", "B"]
|
||||
|
|
@ -744,7 +744,7 @@ class TestProcessOAInspireParcelAssignment:
|
|||
[
|
||||
[20, 50], # postcode A
|
||||
[30, 50], # postcode A (majority)
|
||||
[80, 50], # postcode B (minority — would be dropped pre-fix)
|
||||
[80, 50], # postcode B (minority, would be dropped pre-fix)
|
||||
]
|
||||
)
|
||||
postcodes = ["A", "A", "B"]
|
||||
|
|
@ -762,7 +762,7 @@ class TestProcessOAInspireParcelAssignment:
|
|||
|
||||
class TestProcessOASeedFootprintGuarantee:
|
||||
"""Every postcode with a UPRN seed in a multi-postcode OA must produce a
|
||||
fragment, even when its partition cell collapses below MIN_GEOM_AREA — an
|
||||
fragment, even when its partition cell collapses below MIN_GEOM_AREA. An
|
||||
active postcode must never be dropped (validate_outputs is zero-tolerance)."""
|
||||
|
||||
def test_collapsed_voronoi_cells_rescued_as_footprints(self):
|
||||
|
|
@ -935,7 +935,7 @@ class TestToWgs84Geojson:
|
|||
to_bng = pyproj.Transformer.from_crs(
|
||||
"EPSG:4326", "EPSG:27700", always_xy=True
|
||||
)
|
||||
# 0.9m x 0.9m square: area 0.81 m², perimeter 3.6 m — pointlike, yet
|
||||
# 0.9m x 0.9m square: area 0.81 m², perimeter 3.6 m. Pointlike, yet
|
||||
# large enough (~8 output-grid cells) to survive the 1e-6 deg snap.
|
||||
tiny = box(530000, 180000, 530000.9, 180000.9)
|
||||
from .output import _snap_to_wgs84_geojson
|
||||
|
|
@ -969,7 +969,7 @@ class TestToWgs84Geojson:
|
|||
|
||||
def test_thin_sliver_keeps_minimal_buffer(self):
|
||||
"""A genuine elongated sliver still carries length, so it is NOT inflated
|
||||
to building scale — only truly pointlike inputs are."""
|
||||
to building scale. Only truly pointlike inputs are."""
|
||||
import pyproj
|
||||
from shapely.geometry import LineString, shape
|
||||
from shapely.ops import transform as transform_geometry
|
||||
|
|
@ -1052,7 +1052,7 @@ class TestFillHoles:
|
|||
assert result.area == pytest.approx(Polygon(outer).area)
|
||||
|
||||
def test_large_hole_kept(self):
|
||||
"""A large (>=1000 m²) hole is likely a real enclosed postcode — keep it."""
|
||||
"""A large (>=1000 m²) hole is likely a real enclosed postcode, so keep it."""
|
||||
outer = [(0, 0), (100, 0), (100, 100), (0, 100), (0, 0)]
|
||||
hole = [(20, 20), (80, 20), (80, 80), (20, 80), (20, 20)] # 60x60 = 3600 m²
|
||||
poly_with_hole = Polygon(outer, [hole])
|
||||
|
|
@ -1260,7 +1260,7 @@ class TestKeepDetachedParts:
|
|||
instead of dropping all but the largest, which left ~1.8% uncovered gaps."""
|
||||
|
||||
def test_far_apart_parts_both_kept(self):
|
||||
# Two 50x50m blocks 30m apart — wider than the 10m merge buffer.
|
||||
# Two 50x50m blocks 30m apart, wider than the 10m merge buffer.
|
||||
a = box(0, 0, 50, 50) # 2500 m²
|
||||
b = box(80, 0, 130, 50) # 2500 m², 30m gap
|
||||
geom = merge_fragments([("AA1 1AA", a), ("AA1 1AA", b)])["AA1 1AA"]
|
||||
|
|
@ -1421,7 +1421,7 @@ class TestGeojsonGeometrySliverValidity:
|
|||
class TestColocatedPostcodesAllRetained:
|
||||
"""Co-located non-geographic postcodes (e.g. AL1 9xx) have heavily-overlapping
|
||||
tiny footprints; the de-overlap pass trims most to sub-grid slivers. None may
|
||||
be dropped — every active postcode must keep a (valid, non-empty) boundary."""
|
||||
be dropped: every active postcode must keep a (valid, non-empty) boundary."""
|
||||
|
||||
def test_overlapping_tiny_footprints_none_dropped(self, tmp_path):
|
||||
from shapely.geometry import Point, shape
|
||||
|
|
@ -1447,7 +1447,7 @@ class TestSafeOverlayHelpers:
|
|||
"""The robust overlay helpers retry on a fixed-precision grid after a
|
||||
GEOSException (e.g. ``side location conflict`` from near-coincident OA-seam
|
||||
edges). The grid is a caller-supplied parameter: metres for the BNG stages,
|
||||
1e-6 degrees for the WGS84 output stage — so the same helper serves both
|
||||
1e-6 degrees for the WGS84 output stage, so the same helper serves both
|
||||
without crushing degree-scale shapes on the metre default."""
|
||||
|
||||
def test_grid_param_honored_on_clean_inputs(self):
|
||||
|
|
@ -1497,7 +1497,7 @@ class TestSafeOverlayHelpers:
|
|||
|
||||
# A self-intersecting bow-tie: invalid. set_precision()'s DEFAULT
|
||||
# ('valid_output') mode runs its own noding pass that re-raises
|
||||
# 'side location conflict' on this — which is exactly how the production
|
||||
# 'side location conflict' on this, which is exactly how the production
|
||||
# crash happened (the fallback re-raised the error it was meant to absorb).
|
||||
_BOWTIE = Polygon([(0, 0), (1e-5, 1e-5), (1e-5, 0), (0, 1e-5), (0, 0)])
|
||||
# make_valid() of this spiky ring returns a mixed-dimension
|
||||
|
|
@ -1831,7 +1831,7 @@ class TestFragmentsCache:
|
|||
def test_missing_input_is_ignored(self, tmp_path):
|
||||
cache = tmp_path / "fragments_cache.parquet"
|
||||
cache.write_text("c")
|
||||
# arcgis is optional/absent — it cannot have invalidated the cache.
|
||||
# arcgis is optional/absent, so it cannot have invalidated the cache.
|
||||
assert fragments_cache_is_fresh(cache, [tmp_path / "absent.parquet"]) is True
|
||||
|
||||
|
||||
|
|
@ -1885,7 +1885,7 @@ class TestEliminateSmallDetachedParts:
|
|||
assert after == pytest.approx(before, rel=1e-6), "coverage must be conserved"
|
||||
|
||||
def test_genuine_large_split_is_kept(self):
|
||||
# Two substantial parts (both 40000 m²) far apart — a real bisection, not
|
||||
# Two substantial parts (both 40000 m²) far apart: a real bisection, not
|
||||
# scatter. Neither is below the absolute/fraction thresholds, so both stay.
|
||||
a = MultiPolygon([_mbox(0, 0, 200, 200), _mbox(400, 0, 600, 200)])
|
||||
out = _as_dict(_eliminate_small_detached_parts([("A", a)]))
|
||||
|
|
@ -1909,8 +1909,8 @@ class TestEliminateSmallDetachedParts:
|
|||
assert out["A"].geom_type == "Polygon"
|
||||
|
||||
def test_largest_part_always_retained_no_postcode_dropped(self):
|
||||
# Even a postcode that is ENTIRELY a tiny sliver keeps its (largest) part —
|
||||
# active postcodes must never be dropped (validate_outputs is zero-tolerance).
|
||||
# Even a postcode that is ENTIRELY a tiny sliver keeps its (largest) part.
|
||||
# Active postcodes must never be dropped (validate_outputs is zero-tolerance).
|
||||
tiny = _mbox(500, 500, 505, 505) # 25 m², the whole postcode
|
||||
big = _mbox(0, 0, 300, 300)
|
||||
out = _as_dict(
|
||||
|
|
@ -1961,7 +1961,7 @@ class TestEliminateSmallDetachedParts:
|
|||
assert out["A"].geom_type == "Polygon"
|
||||
|
||||
def test_gapped_sliver_within_nearest_radius_is_reassigned_not_dropped(self):
|
||||
# A 300 m² sliver 1.2 m from its only neighbour N — beyond the border probe
|
||||
# A 300 m² sliver 1.2 m from its only neighbour N: beyond the border probe
|
||||
# but inside the nearest-fallback radius. Before the gather buffer was
|
||||
# widened to the accept radius, the old (smaller) gather buffer never
|
||||
# returned N, so the sliver fell through to the drop branch. Now it is
|
||||
|
|
@ -2017,7 +2017,7 @@ class TestGreenspaceReconnect:
|
|||
from shapely.strtree import STRtree
|
||||
|
||||
postcode = box(0, 0, 200, 100)
|
||||
barrier = box(60, 0, 130, 100) # 70 m wide — beyond 2x the bridge width
|
||||
barrier = box(60, 0, 130, 100) # 70 m wide, beyond 2x the bridge width
|
||||
result = subtract_greenspace(postcode, STRtree([barrier]), [barrier])
|
||||
assert result.geom_type == "MultiPolygon", "wide barrier should stay split"
|
||||
assert len(result.geoms) == 2
|
||||
|
|
@ -2035,7 +2035,7 @@ class TestGreenspaceReconnect:
|
|||
|
||||
def test_result_is_always_polygonal(self):
|
||||
# Regression: _reconnect_split must never return a GeometryCollection
|
||||
# (line/point debris) — downstream to_wgs84_geojson_multi would truncate a
|
||||
# (line/point debris). Downstream to_wgs84_geojson_multi would truncate a
|
||||
# GC to a single part, silently dropping substantial pieces. Sweep many
|
||||
# strip widths/offsets (incl. coincident-edge-prone integer geometries).
|
||||
from shapely.strtree import STRtree
|
||||
|
|
|
|||
|
|
@ -22,13 +22,13 @@ def compute_voronoi_regions(
|
|||
points = points.astype(np.float64)
|
||||
|
||||
# Deduplicate points, keeping one per (location, postcode) pair. Coords are
|
||||
# rounded to mm precision for stable hashing — UPRN inputs are already integer
|
||||
# rounded to mm precision for stable hashing. UPRN inputs are already integer
|
||||
# metres, but the float64 cast can introduce ULP noise.
|
||||
#
|
||||
# Where several DISTINCT postcodes share one coordinate, jitter ALL of them
|
||||
# onto a small regular polygon (equal 0.01m radius, equally spaced by angle)
|
||||
# so their Voronoi cells become equal wedges and NONE is crushed. Leaving any
|
||||
# seed at the centre — or innermost on a spiral — squeezes its cell below
|
||||
# seed at the centre (or innermost on a spiral) squeezes its cell below
|
||||
# MIN_GEOM_AREA, which _clean_polygonal then drops downstream, silently losing
|
||||
# an active postcode. Seeds at a UNIQUE coordinate are left exactly on their
|
||||
# UPRN (no perturbation of normal Voronoi output). Coords are rounded to mm
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue