Compare commits

..

5 commits

Author SHA1 Message Date
408d3f87f0 Fix CI
All checks were successful
Build and publish Docker image / build-and-push (push) Successful in 9m50s
CI / Check (push) Successful in 13m54s
2026-06-14 21:21:22 +01:00
e90a3285a1 Fmt 2026-06-14 15:14:43 +01:00
083f8a982e Small fixes 2026-06-14 14:52:44 +01:00
54fbcb1ea6 Add one change & crime 2026-06-14 14:51:29 +01:00
8e4c56bb0d Cache postcodes 2026-06-14 14:50:38 +01:00
51 changed files with 2328 additions and 382 deletions

View file

@ -29,6 +29,11 @@ jobs:
- name: Install frontend dependencies
working-directory: frontend
# Chrome isn't needed for these checks (lint/typecheck/vitest-jsdom), so
# skip puppeteer's postinstall browser download — it's slow and a flaky
# point of failure. The prerender build installs Chrome explicitly.
env:
PUPPETEER_SKIP_DOWNLOAD: "true"
run: npm ci
- name: Install screenshot service dependencies

View file

@ -12,10 +12,19 @@ ENV BUGSINK_ENVIRONMENT=$BUGSINK_ENVIRONMENT
ENV BUGSINK_RELEASE=$BUGSINK_RELEASE
ENV BUGSINK_SEND_DEFAULT_PII=$BUGSINK_SEND_DEFAULT_PII
# Puppeteer's bundled Chrome downloads from storage.googleapis.com, which is
# geo-blocked on the CI host ("this service is not available in your location").
# Install Chromium from Debian instead: it's reachable, currently tracks the same
# Chrome 149.x puppeteer expects, and apt pulls in the system libraries it needs.
# So: skip puppeteer's postinstall browser download and point its launcher (used
# by the prerender step) at the system binary.
ENV PUPPETEER_SKIP_DOWNLOAD=true \
PUPPETEER_EXECUTABLE_PATH=/usr/bin/chromium
COPY frontend/package.json frontend/package-lock.json ./
RUN npm ci
RUN apt-get update \
&& npx puppeteer browsers install chrome --install-deps \
&& apt-get install -y --no-install-recommends chromium \
&& rm -rf /var/lib/apt/lists/*
COPY frontend/ ./
RUN npm run build

224
finder/README.md Normal file
View file

@ -0,0 +1,224 @@
# Finder — property listing scraper
Scrapes Greater-London sale listings from **Rightmove**, **OnTheMarket**, and
**Zoopla**, recovers each property's true full postcode, and writes a single
parquet (`data/online_listings_buy.parquet`) that the rest of the app consumes
(after a separate enrich step — see [Output](#output)).
`main.py` is the only entry point; everything else is library code.
---
## How it works (and why it's careful about postcodes)
Every portal's **search** API exposes only an *outcode*-level address (e.g.
`"…, London, SW9"`) plus map coordinates — never the full unit postcode. The
full postcode lives on each listing's **detail page**, so the scraper fetches
detail pages to recover it, and only trusts a detail postcode when its outcode
agrees with the coordinate-nearest postcode (so a stale/wrong value can never
silently relocate a listing). When no trustworthy detail postcode is found, it
falls back to the coordinate-nearest postcode. See the module docstrings in
`rightmove.py`, `onthemarket.py`, and `zoopla.py` for the per-portal data model.
Detail fetching is the dominant cost, so it is:
- **cached across runs**`data/detail_cache/{source}.json` maps listing id →
recovered postcode; a re-run only fetches *newly-appeared* listings;
- **fetched concurrently** for the HTTP portals (Rightmove, OnTheMarket), bounded
by a shared global rate limiter so the VPN egress stays polite;
- **gated and capped** per outcode.
See [Performance](#performance--caching).
---
## Prerequisites
The scraper egresses through a VPN. There are two supported ways to provide it:
- **Shared netns (compose, recommended):** an **external `media_gluetun`**
container (qmcgaw/gluetun) must already be running on the host. It is managed
by a *different* compose; `finder/docker-compose.yml` attaches to its network
namespace via `network_mode: "container:media_gluetun"`.
- **HTTP proxy (standalone):** reach a Gluetun HTTP proxy at `GLUETUN_PROXY`
(default `http://gluetun:8888`), or set `GLUETUN_PROXY=""` for a direct,
un-tunnelled connection.
Also required: the ARCGIS postcode parquet at `../property-data/arcgis_data.parquet`
(override with `ARCGIS_PATH`).
---
## Running
### Docker Compose (recommended — the only way that does Zoopla)
`finder/docker-compose.yml` brings up the scraper plus **FlareSolverr** (which
solves Zoopla's Cloudflare challenge), both sharing `media_gluetun`'s netns. This
is the intended production-like path.
```bash
cd finder
# Start the sidecars (finder stays up via `sleep infinity`).
docker compose up -d --build flaresolverr finder
# Run scrapes inside the container (uv run uses the image's /opt/venv):
docker compose exec finder uv run python main.py --source all
docker compose exec finder uv run python main.py --source zoopla --outcodes SW9 --test
docker compose down
```
> If a leftover `finder_flaresolverr` container exists from earlier testing,
> remove it first: `docker rm -f finder_flaresolverr`.
In this setup `GLUETUN_PROXY=""` (the shared netns already tunnels everything),
`ZOOPLA_FETCHER=flaresolverr`, and `DATA_DIR` / `ARCGIS_PATH` are preset by the
compose file.
### Standalone (quick Rightmove / OnTheMarket dev runs)
Zoopla needs FlareSolverr, so standalone is for the HTTP portals. You just need
a venv and VPN reachability.
```bash
cd finder
# One-time: create the venv from the lockfile.
uv sync --frozen # creates .venv with httpx, polars, fake-useragent, …
# Small, safe run into a temp dir (does NOT touch real data/):
.venv/bin/python main.py --source rightmove --outcodes SW9 \
--max-properties-per-source 20 --output-dir /tmp/finder-smoke
# Go direct instead of via the gluetun proxy hostname:
GLUETUN_PROXY="" .venv/bin/python main.py --source onthemarket --outcodes SW9 \
--output-dir /tmp/finder-smoke
```
(`uv run python main.py …` works too and resolves the env automatically.)
---
## CLI reference (`main.py`)
| Flag | Default | Meaning |
|------|---------|---------|
| `--source rightmove,onthemarket` | `all` | Comma-separated portal(s): any of `rightmove`, `onthemarket`, `zoopla`, or `all`. |
| `--outcodes SW9,E14,BR1` | — | Specific outcodes (must be Greater-London-ish). Otherwise the full London set is loaded from ARCGIS. |
| `--limit-outcodes N` | — | Cap the number of outcodes (quick smoke). |
| `--max-properties-per-source N` | — | Stop each source after N transformed listings. |
| `--output-dir DIR` | `data/` | Where the parquet (and `detail_cache/`) are written. |
| `--test` | off | ~10 likely-London outcodes, ≤100 listings/source, writes to `data/test/`. |
> **Always pass `--output-dir /tmp/...` for testing** — the default `data/` holds
> the real listings the app consumes.
### Stopping a run
`Ctrl+C` (SIGINT) — or `docker stop` (SIGTERM) — triggers a **graceful
shutdown**: every source stops at its next outcode boundary, in-flight delays
and retry backoffs wake immediately, and the run still persists the detail
caches and writes the listings collected so far before exiting (code `130`).
Press `Ctrl+C` a second time to force-quit. See `shutdown.py`.
---
## Sources & what each needs
| Source | Transport | Needs FlareSolverr? | Concurrency | Notes |
|--------|-----------|---------------------|-------------|-------|
| Rightmove | plain httpx | no | concurrent detail fetches | main path |
| OnTheMarket | plain httpx | no | concurrent detail fetches | `__NEXT_DATA__` JSON |
| Zoopla | browser / FlareSolverr | **yes** (`ZOOPLA_FETCHER=flaresolverr`, default) | serial (browser-bound) | Cloudflare-protected; skipped gracefully if FlareSolverr is unavailable |
Rightmove and OnTheMarket run **concurrently in worker threads**; Zoopla runs on
the **main thread** (its per-outcode wall-clock guard uses `SIGALRM`, which only
fires on the main thread). One source failing never kills the others.
---
## Output
Each run writes `<output-dir>/online_listings_buy.parquet`.
A **separate enrich step** (outside `finder/`) turns that into
`online_listings_buy_enriched.parquet`, which is what the Rust backend actually
loads (`--actual-listings-path …/online_listings_buy_enriched.parquet` in the
top-level `docker-compose.yml`). That enrich/scheduling pipeline is **not**
documented here — only the raw scrape is.
The top-level `docker-compose.yml` (Rust `server`, `frontend`, `pocketbase`,
`screenshot`) is the **web app**; it is downstream of the scrape and is **not**
required to run the scraper.
---
## Performance & caching
| Mechanism | Where | Effect |
|-----------|-------|--------|
| **Persistent detail cache** | `data/detail_cache/{source}.json` | A listing's postcode never changes, so a re-run reuses cached results and only fetches new listings. Delete this folder to force a full re-fetch. |
| **Concurrent detail fetches** | Rightmove, OnTheMarket | Detail pages fetched in parallel instead of one-at-a-time. |
| **Global rate limiter** | `http_client.RATE_LIMITER` | Caps the *combined* request rate across all threads/portals so concurrency stays polite. |
> **Note on the "accurate-pin skip" flag (`RIGHTMOVE_SKIP_DETAILS_FOR_ACCURATE_PINS`):**
> it is currently a **no-op**. The idea was to skip the detail fetch for listings
> the search already pins precisely (`location.pinType == "ACCURATE_POINT"`), but
> Rightmove's live search API does not include `pinType` in the payload (only
> `latitude`/`longitude`), so nothing is ever skipped. It degrades safely (no
> accuracy loss) but provides no speed-up today.
---
## Configuration
Environment variables (override the defaults in `constants.py`):
| Variable | Default | Purpose |
|----------|---------|---------|
| `DATA_DIR` | `finder/data` | Output root. |
| `ARCGIS_PATH` | `../property-data/arcgis_data.parquet` | Postcode reference data. |
| `GLUETUN_PROXY` | `http://gluetun:8888` | HTTP proxy for egress; `""` = direct. |
| `GLUETUN_CONTROL_URL` | `http://gluetun:8000` | Gluetun control API. |
| `FLARESOLVERR_URL` | `http://gluetun:8191/v1` | FlareSolverr endpoint (Zoopla). |
| `ZOOPLA_FETCHER` | `flaresolverr` | `flaresolverr` or `camoufox`. |
| `ZOOPLA_OUTCODE_TIMEOUT_SECONDS` | `300` | Per-outcode wall-clock budget for Zoopla. |
| `DETAIL_FETCH_CONCURRENCY` | `8` | Parallel detail fetches (Rightmove/OTM). |
| `REQUESTS_PER_SECOND` | `10` | Global request-rate cap. Lower it if you see `429`/`403`. |
| `RIGHTMOVE_SKIP_DETAILS_FOR_ACCURATE_PINS` | `1` | Inert today (see note above). |
Non-env code constants worth knowing (`constants.py` / `onthemarket.py`):
`RIGHTMOVE_FETCH_DETAILS`, `RIGHTMOVE_MAX_DETAILS_PER_OUTCODE` (4000),
`OTM_FETCH_DETAILS`, `OTM_MAX_DETAILS_PER_OUTCODE` (400),
`ZOOPLA_FETCH_DETAILS`, `ZOOPLA_MAX_DETAILS_PER_OUTCODE` (4000).
---
## Tests
`pytest` is not a declared dependency; run it ephemerally with uv (no project
change needed):
```bash
cd finder
uv run --with pytest pytest -q
```
---
## Repo layout
| File | Responsibility |
|------|----------------|
| `main.py` | CLI entry point: parse args, build the postcode index, call `run_scrape`. |
| `scraper.py` | Orchestration: per-source runners, provider parallelism, cache load/save, merge + write. |
| `rightmove.py` / `onthemarket.py` / `zoopla.py` | Per-portal search + detail scraping and parsing. |
| `transform.py` | Raw listing → output schema; postcode trust rules. |
| `http_client.py` | Shared httpx client, retry/backoff, and the global `RATE_LIMITER`. |
| `postcode_cache.py` | Persistent (cross-run) detail-cache load/save. |
| `spatial.py` | Grid spatial index for coordinate → nearest postcode. |
| `storage.py` | Parquet writer (server-ready column names). |
| `constants.py` | Tunables and endpoints. |
| `test_*.py` | Unit tests. |

View file

@ -15,6 +15,16 @@ DELAY_BETWEEN_PAGES = 0.3
DELAY_BETWEEN_OUTCODES = 0.5
MAX_RETRIES = 3
RETRY_BASE_DELAY = 2.0
# Concurrency + global rate limiting for the HTTP-based scrapers (Rightmove and
# OnTheMarket). Detail-page fetches dominate runtime and are independent,
# idempotent GETs, so they are fetched concurrently rather than one-at-a-time.
# A single shared token-bucket limiter (http_client.RATE_LIMITER) caps the
# COMBINED request rate across every worker thread and both providers, so the
# VPN egress IP stays polite no matter how high the concurrency is. Tune both
# down if the portals start returning 429/403.
DETAIL_FETCH_CONCURRENCY = int(os.environ.get("DETAIL_FETCH_CONCURRENCY", "8"))
REQUESTS_PER_SECOND = float(os.environ.get("REQUESTS_PER_SECOND", "10"))
GRID_CELL_SIZE = 0.01 # degrees for postcode spatial index
MAX_BEDROOMS = 20 # sanity cap — values above this are almost certainly parsing errors
@ -35,6 +45,21 @@ RIGHTMOVE_DETAIL_URL = "https://www.rightmove.co.uk/properties/{id}"
RIGHTMOVE_FETCH_DETAILS = True # fetch detail pages for true per-listing postcodes
RIGHTMOVE_MAX_DETAILS_PER_OUTCODE = 4000 # max detail-page fetches per outcode
# Skip the detail fetch when the search result already pins the property
# precisely. Each search-result `location` carries a `pinType`: for
# "ACCURATE_POINT" listings the coordinates are rooftop-exact, so the
# coordinate-nearest postcode already has the right outcode (and almost always
# the right unit) and the detail page adds little. The detail fetch earns its
# keep on APPROXIMATE pins (new-builds/developments) where Rightmove
# deliberately fuzzes the coordinates. Degrades safely: when `pinType` is absent
# from the search payload, nothing is skipped (behaviour is unchanged), so this
# is only a speed-up to the extent the field is present — verify against a live
# search response before relying on the saving.
RIGHTMOVE_SKIP_DETAILS_FOR_ACCURATE_PINS = (
os.environ.get("RIGHTMOVE_SKIP_DETAILS_FOR_ACCURATE_PINS", "1") != "0"
)
RIGHTMOVE_ACCURATE_PIN_TYPE = "ACCURATE_POINT"
# OnTheMarket
ONTHEMARKET_BASE = "https://www.onthemarket.com"

View file

@ -1,14 +1,56 @@
import logging
import random
import threading
import time
import httpx
from fake_useragent import UserAgent
from constants import GLUETUN_PROXY, MAX_RETRIES, RETRY_BASE_DELAY
import shutdown
from constants import (
GLUETUN_PROXY,
MAX_RETRIES,
REQUESTS_PER_SECOND,
RETRY_BASE_DELAY,
)
log = logging.getLogger("rightmove")
class RateLimiter:
"""Thread-safe global limiter: spaces request starts by a minimum interval.
Detail-page fetches run concurrently across many worker threads (and across
providers), but a single shared limiter caps their COMBINED rate so the VPN
egress IP stays polite. Each ``acquire()`` reserves the next free time slot
under a lock, then sleeps (outside the lock) until that slot so N threads
calling concurrently are spaced ``1/rate_per_second`` apart rather than all
firing at once. ``rate_per_second <= 0`` disables limiting."""
def __init__(self, rate_per_second: float):
self._interval = 1.0 / rate_per_second if rate_per_second > 0 else 0.0
self._lock = threading.Lock()
self._next = 0.0
def acquire(self) -> None:
if self._interval <= 0:
return
with self._lock:
now = time.monotonic()
if now >= self._next:
self._next = now + self._interval
wait = 0.0
else:
wait = self._next - now
self._next += self._interval
if wait > 0:
shutdown.sleep(wait)
# Shared by every HTTP-based fetch (search pages and detail pages, all
# providers). Spacing is global, so politeness is decoupled from concurrency.
RATE_LIMITER = RateLimiter(REQUESTS_PER_SECOND)
_ua = UserAgent(
browsers=["Chrome", "Edge"], os=["Windows", "Mac OS X"], min_version=120.0
)
@ -33,7 +75,10 @@ def fetch_with_retry(
compatibility with older callers; 403 is now treated as non-retryable.
"""
for attempt in range(MAX_RETRIES):
if shutdown.stop_requested():
return None
try:
RATE_LIMITER.acquire()
resp = client.get(url, params=params)
if resp.status_code == 200:
return resp.json()
@ -50,7 +95,7 @@ def fetch_with_retry(
MAX_RETRIES,
delay,
)
time.sleep(delay)
shutdown.sleep(delay)
continue
log.error("HTTP %d from %s (non-retryable)", resp.status_code, url)
return None
@ -69,6 +114,6 @@ def fetch_with_retry(
MAX_RETRIES,
delay,
)
time.sleep(delay)
shutdown.sleep(delay)
log.error("All %d retries exhausted for %s", MAX_RETRIES, url)
return None

View file

@ -5,10 +5,12 @@ import tempfile
import time
from pathlib import Path
import shutdown
from constants import DATA_DIR, REPO_DIR
SOURCE_CHOICES = ("rightmove", "onthemarket", "zoopla", "all")
SOURCES = ("rightmove", "onthemarket", "zoopla")
SOURCE_CHOICES = (*SOURCES, "all")
TEST_MAX_PROPERTIES_PER_SOURCE = 100
TEST_OUTCODES = (
"E1",
@ -47,9 +49,13 @@ def parse_args() -> argparse.Namespace:
)
parser.add_argument(
"--source",
choices=SOURCE_CHOICES,
default="all",
help="Portal to scrape. 'all' runs Rightmove, OnTheMarket, and Zoopla.",
metavar="SOURCES",
help=(
"Comma-separated portal(s) to scrape: any of "
f"{', '.join(SOURCES)}, or 'all' (default). "
"E.g. --source rightmove,onthemarket."
),
)
parser.add_argument(
"--output-dir",
@ -100,15 +106,35 @@ def configure_logging() -> None:
def selected_sources(source: str) -> list[str]:
if source == "all":
return ["rightmove", "onthemarket", "zoopla"]
return [source]
"""Resolve --source: one or more comma-separated portals, or 'all'.
Accepts e.g. ``rightmove,onthemarket`` (whitespace and case tolerant).
Unknown values are rejected; the result is deduplicated and returned in the
canonical ``SOURCES`` order so downstream merge/dedup stays deterministic."""
requested = [part.strip().lower() for part in source.split(",")]
requested = [part for part in requested if part]
if not requested:
raise SystemExit("--source was empty")
unknown = sorted(set(requested) - set(SOURCE_CHOICES))
if unknown:
raise SystemExit(
f"Unknown --source value(s): {', '.join(unknown)}. "
f"Choose from {', '.join(SOURCE_CHOICES)} (comma-separated)."
)
if "all" in requested:
return list(SOURCES)
return [portal for portal in SOURCES if portal in requested]
def main() -> int:
args = parse_args()
configure_standalone_runtime()
configure_logging()
# Ctrl+C (and SIGTERM, e.g. `docker stop`) asks the scrapers to wind down
# gracefully — each source stops at its next outcode boundary and the run
# still persists detail caches and writes the listings collected so far.
shutdown.install_signal_handlers()
if args.limit_outcodes is not None and args.limit_outcodes < 1:
raise SystemExit("--limit-outcodes must be greater than zero")
@ -182,6 +208,14 @@ def main() -> int:
)
elapsed = time.monotonic() - started
if shutdown.stop_requested():
log.warning(
"Scrape interrupted after %.1fs; partial results written to %s",
elapsed,
result.get("path"),
)
log.info("Result: %s", result)
return 130 # 128 + SIGINT, the conventional Ctrl+C exit code.
log.info("Scrape finished in %.1fs", elapsed)
log.info("Result: %s", result)
if args.test and result.get("errors"):

View file

@ -40,17 +40,19 @@ import json
import logging
import random
import re
import time
from concurrent.futures import ThreadPoolExecutor
import httpx
import shutdown
from constants import (
DELAY_BETWEEN_PAGES,
DETAIL_FETCH_CONCURRENCY,
MAX_BEDROOMS,
MAX_RETRIES,
ONTHEMARKET_BASE,
RETRY_BASE_DELAY,
)
from http_client import RATE_LIMITER
from spatial import PostcodeSpatialIndex
from transform import (
clean_listing_address,
@ -89,10 +91,24 @@ _HTML_HEADERS = {
# listingId -> recovered full postcode (or None). Failures are cached too so a
# broken or postcode-less detail page is not re-fetched within a run (the same
# listing can reappear across overlapping outcode searches).
# listing can reappear across overlapping outcode searches). Seeded from /
# dumped to a persistent on-disk cache by the orchestrator (see
# postcode_cache.py) so a recurring scrape only re-fetches newly-listed
# properties rather than every listing every run.
_detail_postcode_cache: dict[str, str | None] = {}
def seed_detail_cache(mapping: dict) -> None:
"""Pre-populate the in-memory detail cache from a persisted snapshot."""
if mapping:
_detail_postcode_cache.update(mapping)
def detail_cache_snapshot() -> dict:
"""Return a JSON-serialisable copy of the in-memory detail cache."""
return dict(_detail_postcode_cache)
def _fetch_page_json(client: httpx.Client, outcode: str, page_num: int) -> dict | None:
"""GET one search-results page and return the embedded __NEXT_DATA__ JSON.
@ -103,7 +119,10 @@ def _fetch_page_json(client: httpx.Client, outcode: str, page_num: int) -> dict
params = {"page": str(page_num)} if page_num > 1 else None
for attempt in range(MAX_RETRIES):
if shutdown.stop_requested():
return None
try:
RATE_LIMITER.acquire()
resp = client.get(
url,
params=params,
@ -121,7 +140,7 @@ def _fetch_page_json(client: httpx.Client, outcode: str, page_num: int) -> dict
"%s from %s, retry %d/%d in %.1fs",
type(exc).__name__, url, attempt + 1, MAX_RETRIES, delay,
)
time.sleep(delay)
shutdown.sleep(delay)
continue
if 300 <= resp.status_code < 400:
@ -151,7 +170,7 @@ def _fetch_page_json(client: httpx.Client, outcode: str, page_num: int) -> dict
"HTTP %d from %s, retry %d/%d in %.1fs",
resp.status_code, url, attempt + 1, MAX_RETRIES, delay,
)
time.sleep(delay)
shutdown.sleep(delay)
continue
log.error("HTTP %d from %s (non-retryable)", resp.status_code, url)
return None
@ -218,7 +237,8 @@ def _fetch_detail_postcode(
reappears across overlapping outcode searches is fetched at most once. Plain
HTTPS GET OnTheMarket detail pages have no Cloudflare challenge. Network /
parse errors degrade gracefully to None so the caller falls back to the
coordinate-nearest postcode.
coordinate-nearest postcode. Safe to call concurrently: distinct listing ids
write distinct cache keys, and the shared RATE_LIMITER spaces the GETs.
"""
if listing_id in _detail_postcode_cache:
return _detail_postcode_cache[listing_id]
@ -231,7 +251,10 @@ def _fetch_detail_postcode(
result: str | None = None
if full_url:
for attempt in range(MAX_RETRIES):
if shutdown.stop_requested():
break
try:
RATE_LIMITER.acquire()
resp = client.get(
full_url, headers=_HTML_HEADERS, follow_redirects=True
)
@ -246,7 +269,7 @@ def _fetch_detail_postcode(
"%s from %s, retry %d/%d in %.1fs",
type(exc).__name__, full_url, attempt + 1, MAX_RETRIES, delay,
)
time.sleep(delay)
shutdown.sleep(delay)
continue
if resp.status_code == 200:
@ -258,7 +281,7 @@ def _fetch_detail_postcode(
"HTTP %d from %s, retry %d/%d in %.1fs",
resp.status_code, full_url, attempt + 1, MAX_RETRIES, delay,
)
time.sleep(delay)
shutdown.sleep(delay)
continue
log.debug(
"OnTheMarket detail %s returned HTTP %d (no postcode)",
@ -422,6 +445,47 @@ def transform_property(
}
def _prime_detail_postcodes(
client: httpx.Client,
raw_listings: list[dict],
detail_cap: int,
) -> None:
"""Fill ``_detail_postcode_cache`` for the listings that need a detail page.
Picks the fresh (uncached) listings up to ``detail_cap`` per outcode then
fetches their detail pages CONCURRENTLY, bounded by
``DETAIL_FETCH_CONCURRENCY`` (the shared RATE_LIMITER keeps the combined
request rate polite). Cached listings cost neither a slot nor a GET. The
worklist is deduplicated, so distinct ids write distinct cache keys and the
concurrent fetches never race on the same key."""
if not OTM_FETCH_DETAILS or detail_cap <= 0:
return
worklist: list[tuple[str, str]] = [] # (details_url, listing_id)
seen: set[str] = set()
for raw in raw_listings:
listing_id = str(raw.get("id") or "")
if not listing_id or listing_id in seen:
continue
seen.add(listing_id)
if listing_id in _detail_postcode_cache:
continue
worklist.append((raw.get("details-url") or "", listing_id))
if len(worklist) >= detail_cap:
break
if not worklist:
return
workers = min(DETAIL_FETCH_CONCURRENCY, len(worklist))
with ThreadPoolExecutor(max_workers=workers) as pool:
# Drain the iterator so every fetch runs and populates the cache here.
for _ in pool.map(
lambda item: _fetch_detail_postcode(client, item[0], item[1]), worklist
):
pass
def search_outcode(
client: httpx.Client,
outcode: str,
@ -430,17 +494,19 @@ def search_outcode(
) -> list[dict]:
"""Paginate through OnTheMarket sale results for one outcode.
When ``OTM_FETCH_DETAILS`` is enabled, up to
``OTM_MAX_DETAILS_PER_OUTCODE`` listings per outcode have their detail page
fetched for the property's own postcode (see ``_fetch_detail_postcode``);
the rest fall back to the coordinate-nearest postcode.
Search pages are paginated serially (the RATE_LIMITER spaces the GETs); then,
when ``OTM_FETCH_DETAILS`` is enabled, up to ``OTM_MAX_DETAILS_PER_OUTCODE``
listings per outcode have their detail page fetched CONCURRENTLY for the
property's own postcode (see ``_fetch_detail_postcode``); the rest fall back
to the coordinate-nearest postcode.
"""
properties: list[dict] = []
raw_listings: list[dict] = []
seen_ids: set[str] = set()
page_num = 1
details_fetched = 0
while True:
if shutdown.stop_requested():
break
data = _fetch_page_json(client, outcode, page_num)
if data is None:
break
@ -453,47 +519,45 @@ def search_outcode(
)
break
raw_listings = state.get("list") or []
if not raw_listings:
page_listings = state.get("list") or []
if not page_listings:
break
for raw in raw_listings:
for raw in page_listings:
listing_id = str(raw.get("id") or "")
if listing_id and listing_id in seen_ids:
continue
seen_ids.add(listing_id)
raw_listings.append(raw)
detail_postcode = None
if OTM_FETCH_DETAILS and listing_id:
# Cached lookups are free; only fresh GETs count toward the cap
# and incur the inter-request delay.
cached = listing_id in _detail_postcode_cache
if cached or details_fetched < OTM_MAX_DETAILS_PER_OUTCODE:
detail_postcode = _fetch_detail_postcode(
client, raw.get("details-url") or "", listing_id
)
if not cached:
details_fetched += 1
time.sleep(DELAY_BETWEEN_PAGES)
try:
transformed = transform_property(raw, pc_index, detail_postcode)
except Exception as exc:
log.warning(
"OnTheMarket %s property %s failed to transform: %s",
outcode, listing_id or "?", exc,
)
continue
if transformed:
properties.append(transformed)
if max_properties is not None and len(properties) >= max_properties:
return properties
if max_properties is not None and len(raw_listings) >= max_properties:
break
pagination = state.get("paginationControls") or {}
if not pagination.get("next"):
break
page_num += 1
time.sleep(DELAY_BETWEEN_PAGES)
_prime_detail_postcodes(client, raw_listings, OTM_MAX_DETAILS_PER_OUTCODE)
properties: list[dict] = []
for raw in raw_listings:
listing_id = str(raw.get("id") or "")
detail_postcode = (
_detail_postcode_cache.get(listing_id) if OTM_FETCH_DETAILS else None
)
try:
transformed = transform_property(raw, pc_index, detail_postcode)
except Exception as exc:
log.warning(
"OnTheMarket %s property %s failed to transform: %s",
outcode, listing_id or "?", exc,
)
continue
if transformed:
properties.append(transformed)
if max_properties is not None and len(properties) >= max_properties:
break
return properties

67
finder/postcode_cache.py Normal file
View file

@ -0,0 +1,67 @@
"""Persistent detail-fetch caches that survive across scrape runs.
Each portal recovers a listing's true postcode (Rightmove/OnTheMarket) or full
geo dict (Zoopla) from its detail page. That value never changes for a given
listing id, yet the in-memory caches are discarded at the end of every run so
each run re-fetches every listing's detail page from scratch. Persisting the
cache to disk means a steady-state run only fetches NEWLY-appeared listings,
typically a small fraction of the market, which is the single biggest saving
for a recurring scrape (and it *reduces* request volume rather than adding to
it).
The stored values are always JSON-serialisable: Rightmove/OnTheMarket cache
``str | None`` postcodes; Zoopla caches a flat dict of primitives (or ``None``).
A cached ``None`` (a listing whose detail page yielded nothing usable) is
persisted too, so a postcode-less listing is not re-fetched every run.
"""
import json
import logging
import os
import tempfile
from pathlib import Path
log = logging.getLogger("rightmove")
def load_cache(path: str | Path) -> dict:
"""Load a persisted detail cache. Returns ``{}`` when absent or unreadable.
A corrupt or non-object file is treated as empty rather than fatal a bad
cache must never block a scrape; the worst case is re-fetching details."""
p = Path(path)
if not p.exists():
return {}
try:
with open(p, encoding="utf-8") as fh:
data = json.load(fh)
except (OSError, ValueError) as exc:
log.warning("Could not read detail cache %s: %s", p, exc)
return {}
if not isinstance(data, dict):
log.warning("Detail cache %s is not a JSON object; ignoring", p)
return {}
log.info("Loaded %d cached detail entries from %s", len(data), p)
return data
def save_cache(path: str | Path, data: dict) -> None:
"""Atomically write a detail cache to disk (temp file + ``os.replace``).
Creates the parent directory if needed. Write failures are logged and
swallowed: persisting the cache is an optimisation, never a hard
requirement, so a read-only disk must not fail an otherwise-good scrape."""
p = Path(path)
try:
p.parent.mkdir(parents=True, exist_ok=True)
fd, tmp = tempfile.mkstemp(dir=str(p.parent), suffix=".tmp")
try:
with os.fdopen(fd, "w", encoding="utf-8") as fh:
json.dump(data, fh)
os.replace(tmp, p)
finally:
if os.path.exists(tmp):
os.unlink(tmp)
log.info("Saved %d cached detail entries to %s", len(data), p)
except OSError as exc:
log.warning("Could not write detail cache %s: %s", p, exc)

View file

@ -1,20 +1,23 @@
import json
import logging
import re
import time
from concurrent.futures import ThreadPoolExecutor
import httpx
import shutdown
from constants import (
DETAIL_FETCH_CONCURRENCY,
PAGE_SIZE,
DELAY_BETWEEN_PAGES,
RIGHTMOVE_ACCURATE_PIN_TYPE,
RIGHTMOVE_DETAIL_URL,
RIGHTMOVE_FETCH_DETAILS,
RIGHTMOVE_MAX_DETAILS_PER_OUTCODE,
RIGHTMOVE_SKIP_DETAILS_FOR_ACCURATE_PINS,
SEARCH_URL,
TYPEAHEAD_URL,
)
from http_client import fetch_with_retry
from http_client import RATE_LIMITER, fetch_with_retry
from spatial import PostcodeSpatialIndex
from transform import extract_full_postcode, normalize_postcode, transform_property
@ -168,16 +171,32 @@ def parse_detail_postcode(html: str) -> str | None:
# listingId -> true full postcode (or None when unavailable). Failures are
# cached too, so a broken/duplicate listing is fetched at most once per run (the
# same listing can reappear across overlapping outcode searches).
# same listing can reappear across overlapping outcode searches). Seeded from /
# dumped to a persistent on-disk cache by the orchestrator (see
# postcode_cache.py) so a recurring scrape only re-fetches newly-listed
# properties rather than every listing every run.
_detail_postcode_cache: dict[str, str | None] = {}
def seed_detail_cache(mapping: dict) -> None:
"""Pre-populate the in-memory detail cache from a persisted snapshot."""
if mapping:
_detail_postcode_cache.update(mapping)
def detail_cache_snapshot() -> dict:
"""Return a JSON-serialisable copy of the in-memory detail cache."""
return dict(_detail_postcode_cache)
def _fetch_detail_postcode(client: httpx.Client, property_id: str) -> str | None:
"""GET a listing detail page and return its true full postcode (or None).
Results (including failures) are cached by listing id. The detail page is a
plain HTML GET no Cloudflare, unlike Zoopla so a single httpx call
suffices; any error degrades gracefully to the coordinate fallback."""
suffices; any error degrades gracefully to the coordinate fallback. Safe to
call concurrently: distinct listing ids write distinct cache keys, and the
shared RATE_LIMITER spaces the GETs."""
if not property_id:
return None
if property_id in _detail_postcode_cache:
@ -186,6 +205,7 @@ def _fetch_detail_postcode(client: httpx.Client, property_id: str) -> str | None
postcode: str | None = None
url = RIGHTMOVE_DETAIL_URL.format(id=property_id)
try:
RATE_LIMITER.acquire()
resp = client.get(url, headers={"Accept": "text/html"})
if resp.status_code == 200:
postcode = parse_detail_postcode(resp.text)
@ -219,53 +239,83 @@ def resolve_outcode_id(client: httpx.Client, outcode: str) -> str | None:
return None
def _detail_postcode_for(
def _needs_detail_fetch(prop: dict) -> bool:
"""Whether a search result still needs a detail-page fetch for its postcode.
Skips listings the search already pins precisely: an "ACCURATE_POINT"
``pinType`` means rooftop-exact coordinates, so the coordinate-nearest
postcode is trustworthy and the detail page would only confirm it. Listings
with an approximate pin or no ``pinType`` field at all still get fetched,
so this degrades safely to the previous behaviour when the search payload
omits ``pinType``."""
if not RIGHTMOVE_SKIP_DETAILS_FOR_ACCURATE_PINS:
return True
loc = prop.get("location") or {}
return loc.get("pinType") != RIGHTMOVE_ACCURATE_PIN_TYPE
def _prime_detail_postcodes(
client: httpx.Client,
prop: dict,
props: list[dict],
fetch_details: bool,
detail_budget: dict,
) -> str | None:
"""Look up a listing's true postcode, honouring the per-outcode fetch cap.
detail_cap: int,
) -> None:
"""Fill ``_detail_postcode_cache`` for the listings that need a detail page.
Cached listings are always served (they cost neither a cap slot nor a GET);
a fresh fetch is made only while ``detail_budget['remaining'] > 0``."""
if not fetch_details:
return None
property_id = str(prop.get("id") or "")
if not property_id:
return None
if property_id in _detail_postcode_cache:
return _detail_postcode_cache[property_id]
if detail_budget["remaining"] <= 0:
return None
detail_budget["remaining"] -= 1
postcode = _fetch_detail_postcode(client, property_id)
time.sleep(DELAY_BETWEEN_PAGES)
return postcode
Picks the fresh (uncached, not-skipped) listings up to ``detail_cap`` per
outcode then fetches their detail pages CONCURRENTLY, bounded by
``DETAIL_FETCH_CONCURRENCY`` (the shared RATE_LIMITER keeps the combined
request rate polite). Cached listings cost neither a slot nor a GET. The
worklist is deduplicated, so distinct ids write distinct cache keys and the
concurrent fetches never race on the same key."""
if not fetch_details or detail_cap <= 0:
return
worklist: list[str] = []
seen: set[str] = set()
for prop in props:
property_id = str(prop.get("id") or "")
if not property_id or property_id in seen:
continue
seen.add(property_id)
if property_id in _detail_postcode_cache:
continue
if not _needs_detail_fetch(prop):
continue
worklist.append(property_id)
if len(worklist) >= detail_cap:
break
if not worklist:
return
workers = min(DETAIL_FETCH_CONCURRENCY, len(worklist))
with ThreadPoolExecutor(max_workers=workers) as pool:
# Drain the iterator so every fetch runs and populates the cache here.
for _ in pool.map(lambda pid: _fetch_detail_postcode(client, pid), worklist):
pass
def _paginate(
def _collect_search_props(
client: httpx.Client,
outcode_id: str,
outcode: str,
channel_cfg: dict,
pc_index: PostcodeSpatialIndex,
max_properties: int | None = None,
fetch_details: bool = False,
detail_cap: int = 0,
) -> tuple[list[dict], int]:
"""Paginate through search results. Returns (properties, result_count).
"""Paginate the search API for one outcode+channel, collecting raw results.
When ``fetch_details`` is set, up to ``detail_cap`` listings per outcode have
their detail page fetched for the property's TRUE full postcode (see
``parse_detail_postcode``); the rest fall back to coordinate-derived
postcodes."""
properties = []
Returns ``(raw_props, result_count)``. Pagination stays serial each page
reveals the next but is cheap relative to detail fetching, and the
RATE_LIMITER spaces the page GETs. Collection stops at ``max_properties`` raw
listings, the end of results, or Rightmove's ``_MAX_INDEX`` page cap."""
raw_props: list[dict] = []
index = 0
result_count = 0
detail_budget = {"remaining": detail_cap}
while True:
if shutdown.stop_requested():
break
params = {
"useLocationIdentifier": "true",
"locationIdentifier": f"OUTCODE^{outcode_id}",
@ -284,37 +334,17 @@ def _paginate(
)
break
raw_props = data.get("properties", [])
if not raw_props:
page_props = data.get("properties", [])
if not page_props:
break
raw_props.extend(page_props)
for prop in raw_props:
try:
detail_postcode = _detail_postcode_for(
client, prop, fetch_details, detail_budget
)
transformed = transform_property(
prop, outcode, pc_index, detail_postcode=detail_postcode
)
except Exception as exc:
log.warning(
"Rightmove %s/%s property %s failed to transform: %s",
outcode,
channel_cfg["channel"],
prop.get("id", "?"),
exc,
)
continue
if transformed:
properties.append(transformed)
if max_properties is not None and len(properties) >= max_properties:
return properties, result_count
# Check if there are more pages
result_count_str = data.get("resultCount", "0")
result_count = int(result_count_str.replace(",", ""))
index += PAGE_SIZE
if max_properties is not None and len(raw_props) >= max_properties:
break
index += PAGE_SIZE
if index >= result_count:
break
if index >= _MAX_INDEX:
@ -327,7 +357,55 @@ def _paginate(
)
break
time.sleep(DELAY_BETWEEN_PAGES)
return raw_props, result_count
def _paginate(
client: httpx.Client,
outcode_id: str,
outcode: str,
channel_cfg: dict,
pc_index: PostcodeSpatialIndex,
max_properties: int | None = None,
fetch_details: bool = False,
detail_cap: int = 0,
) -> tuple[list[dict], int]:
"""Collect search results, recover true postcodes, and transform them.
Search pages are paginated serially; then when ``fetch_details`` is set
up to ``detail_cap`` listings per outcode have their detail page fetched
CONCURRENTLY for the property's TRUE full postcode (see
``parse_detail_postcode``), with listings the search already pins precisely
skipped (see ``_needs_detail_fetch``). The rest fall back to
coordinate-derived postcodes. Returns ``(properties, result_count)``."""
raw_props, result_count = _collect_search_props(
client, outcode_id, outcode, channel_cfg, max_properties
)
_prime_detail_postcodes(client, raw_props, fetch_details, detail_cap)
properties: list[dict] = []
for prop in raw_props:
property_id = str(prop.get("id") or "")
detail_postcode = (
_detail_postcode_cache.get(property_id) if fetch_details else None
)
try:
transformed = transform_property(
prop, outcode, pc_index, detail_postcode=detail_postcode
)
except Exception as exc:
log.warning(
"Rightmove %s/%s property %s failed to transform: %s",
outcode,
channel_cfg["channel"],
prop.get("id", "?"),
exc,
)
continue
if transformed:
properties.append(transformed)
if max_properties is not None and len(properties) >= max_properties:
break
return properties, result_count

View file

@ -3,9 +3,11 @@ import os
import re
import signal
import time
from concurrent.futures import ThreadPoolExecutor
from contextlib import contextmanager
from functools import partial
from pathlib import Path
from typing import Iterable
from typing import Callable, Iterable
import polars as pl
@ -21,8 +23,13 @@ from constants import (
ZOOPLA_MAX_DETAILS_PER_OUTCODE,
)
import onthemarket
import rightmove
import shutdown
import zoopla
from http_client import make_client
from onthemarket import search_outcode as onthemarket_search_outcode
from postcode_cache import load_cache, save_cache
from rightmove import resolve_outcode_id
from rightmove import search_outcode as rightmove_search_outcode
from spatial import PostcodeSpatialIndex
@ -38,6 +45,16 @@ SALE_CHANNEL = CHANNELS[0]
LONDON_AREAS = sorted({prefix.upper() for prefix in LONDON_OUTCODE_PREFIXES})
OUTCODE_RE = re.compile(r"^([A-Z]{1,2}\d[A-Z0-9]?)")
# Per-source modules exposing the persistent detail-cache hooks
# (seed_detail_cache / detail_cache_snapshot). A listing's recovered postcode
# never changes, so the cache is loaded before a run and dumped after it, letting
# a recurring scrape skip the detail fetch for every listing it has already seen.
_CACHE_MODULES = {
"rightmove": rightmove,
"onthemarket": onthemarket,
"zoopla": zoopla,
}
def _arcgis_columns() -> tuple[str, str]:
"""Return postcode and country column names for supported ARCGIS schemas."""
@ -306,6 +323,9 @@ def _scrape_rightmove(
client = make_client()
try:
for outcode in outcodes:
if shutdown.stop_requested():
log.info("Rightmove: shutdown requested, stopping early")
return
if _source_remaining(results, "rightmove", max_properties_per_source) == 0:
log.info("Rightmove cap reached")
return
@ -314,12 +334,12 @@ def _scrape_rightmove(
outcode_id = resolve_outcode_id(client, outcode)
except Exception as exc:
_record_error(errors, "rightmove", outcode, exc)
time.sleep(DELAY_BETWEEN_OUTCODES)
shutdown.sleep(DELAY_BETWEEN_OUTCODES)
continue
if not outcode_id:
log.debug("No Rightmove outcode ID for %s", outcode)
time.sleep(DELAY_BETWEEN_OUTCODES)
shutdown.sleep(DELAY_BETWEEN_OUTCODES)
continue
remaining = _source_remaining(
@ -348,7 +368,7 @@ def _scrape_rightmove(
except Exception as exc:
_record_error(errors, "rightmove", outcode, exc)
time.sleep(DELAY_BETWEEN_OUTCODES)
shutdown.sleep(DELAY_BETWEEN_OUTCODES)
finally:
client.close()
@ -494,6 +514,9 @@ def _scrape_zoopla_flaresolverr(
try:
for outcode in outcodes:
if shutdown.stop_requested():
log.info("Zoopla: shutdown requested, stopping early")
return
remaining = _source_remaining(results, "zoopla", max_properties_per_source)
if remaining == 0:
log.info("Zoopla cap reached")
@ -511,7 +534,7 @@ def _scrape_zoopla_flaresolverr(
log.info("Zoopla %s: +%d", outcode, added)
except Exception as exc: # noqa: BLE001 - one outcode must not kill the run
_record_error(errors, "zoopla", outcode, exc)
time.sleep(DELAY_BETWEEN_OUTCODES)
shutdown.sleep(DELAY_BETWEEN_OUTCODES)
finally:
session.__exit__(None, None, None)
@ -547,6 +570,9 @@ def _scrape_zoopla(
try:
for outcode in outcodes:
if shutdown.stop_requested():
log.info("Zoopla: shutdown requested, stopping early")
return
if _source_remaining(results, "zoopla", max_properties_per_source) == 0:
log.info("Zoopla cap reached")
return
@ -596,7 +622,7 @@ def _scrape_zoopla(
_record_error(errors, "zoopla", outcode, relaunch_exc)
return
time.sleep(DELAY_BETWEEN_OUTCODES)
shutdown.sleep(DELAY_BETWEEN_OUTCODES)
finally:
if detail_page is not None:
try:
@ -616,6 +642,9 @@ def _scrape_onthemarket(
client = make_client()
try:
for outcode in outcodes:
if shutdown.stop_requested():
log.info("OnTheMarket: shutdown requested, stopping early")
return
if (
_source_remaining(results, "onthemarket", max_properties_per_source)
== 0
@ -644,11 +673,57 @@ def _scrape_onthemarket(
except Exception as exc:
_record_error(errors, "onthemarket", outcode, exc)
time.sleep(DELAY_BETWEEN_OUTCODES)
shutdown.sleep(DELAY_BETWEEN_OUTCODES)
finally:
client.close()
def _seed_detail_caches(sources: list[str], cache_dir: Path) -> None:
"""Load each selected source's persisted detail cache into memory."""
for source in sources:
module = _CACHE_MODULES.get(source)
if module is None:
continue
module.seed_detail_cache(load_cache(cache_dir / f"{source}.json"))
def _save_detail_caches(sources: list[str], cache_dir: Path) -> None:
"""Persist each selected source's in-memory detail cache to disk."""
for source in sources:
module = _CACHE_MODULES.get(source)
if module is None:
continue
save_cache(cache_dir / f"{source}.json", module.detail_cache_snapshot())
def _run_sources(
run_zoopla: Callable[[], None] | None,
background_runners: list[tuple[str, Callable[[], None]]],
errors: list[str],
) -> None:
"""Run the HTTP-based providers concurrently while Zoopla runs inline.
Rightmove and OnTheMarket are plain-httpx and thread-safe, so each runs in a
worker thread. Zoopla must stay on the MAIN thread because its per-outcode
wall-clock guard uses SIGALRM, which Python only delivers there. Each source
writes only its own ``results[source]`` list and appends to the shared
``errors`` list (atomic under the GIL), so there is no cross-source data
race. One source raising never kills the others: each failure is recorded
and the remaining sources still finish."""
with ThreadPoolExecutor(max_workers=max(1, len(background_runners))) as pool:
futures = {pool.submit(fn): name for name, fn in background_runners}
if run_zoopla is not None:
try:
run_zoopla()
except Exception as exc: # noqa: BLE001 - one source must not kill the run
_record_error(errors, "zoopla", "*", exc)
for future, name in futures.items():
try:
future.result()
except Exception as exc: # noqa: BLE001 - one source must not kill the run
_record_error(errors, name, "*", exc)
def run_scrape(
outcodes: list[str],
pc_index: PostcodeSpatialIndex,
@ -679,28 +754,46 @@ def run_scrape(
max_properties_per_source,
)
# Persistent detail caches: load before, dump after. A listing's recovered
# postcode never changes, so a recurring scrape only fetches new listings.
cache_dir = output_base / "detail_cache"
_seed_detail_caches(selected_sources, cache_dir)
# Rightmove and OnTheMarket are plain-httpx and run in worker threads;
# Zoopla runs on the main thread (its SIGALRM timeout requires it). They
# share one RATE_LIMITER, so running them at once stays polite.
background_runners: list[tuple[str, Callable[[], None]]] = []
if "rightmove" in selected_sources:
_scrape_rightmove(
selected_outcodes,
pc_index,
results,
errors,
max_properties_per_source,
)
background_runners.append((
"rightmove",
partial(
_scrape_rightmove,
selected_outcodes,
pc_index,
results,
errors,
max_properties_per_source,
),
))
if "onthemarket" in selected_sources:
_scrape_onthemarket(
selected_outcodes,
pc_index,
results,
errors,
max_properties_per_source,
)
background_runners.append((
"onthemarket",
partial(
_scrape_onthemarket,
selected_outcodes,
pc_index,
results,
errors,
max_properties_per_source,
),
))
run_zoopla: Callable[[], None] | None = None
if "zoopla" in selected_sources:
if pc_coords is None:
pc_coords = build_postcode_coords()
_scrape_zoopla(
run_zoopla = partial(
_scrape_zoopla,
selected_outcodes,
pc_index,
pc_coords,
@ -709,6 +802,11 @@ def run_scrape(
max_properties_per_source,
)
try:
_run_sources(run_zoopla, background_runners, errors)
finally:
_save_detail_caches(selected_sources, cache_dir)
merged, source_counts, deduped = _merge_properties(results)
output_path = output_base / "online_listings_buy.parquet"
if merged:

69
finder/shutdown.py Normal file
View file

@ -0,0 +1,69 @@
"""Process-wide cooperative shutdown for the scrapers.
A single :class:`threading.Event` is set the first time the process receives
SIGINT (Ctrl+C) or SIGTERM. Every scrape loop polls :func:`stop_requested` at
its outcode/page boundaries and every blocking delay goes through :func:`sleep`,
which wakes the instant a stop is requested. So Ctrl+C makes each source stop
*starting* new work and unwind through its normal ``finally`` blocks detail
caches are persisted and whatever has been collected so far is still written
instead of hanging until the worker threads happen to finish (the orchestrator's
``ThreadPoolExecutor`` used to block the exit waiting on them) or losing the run
outright.
A second Ctrl+C escalates to a hard :class:`KeyboardInterrupt` for an impatient
operator. SIGALRM is deliberately left untouched: Zoopla's per-outcode
wall-clock guard owns it."""
import logging
import signal
import threading
log = logging.getLogger("finder")
_STOP = threading.Event()
def stop_requested() -> bool:
"""True once a shutdown signal has been received."""
return _STOP.is_set()
def request_stop() -> None:
"""Ask every scrape loop to stop starting new work."""
_STOP.set()
def reset() -> None:
"""Clear the flag — for tests and repeated in-process runs."""
_STOP.clear()
def sleep(seconds: float) -> None:
"""Sleep that wakes immediately once a stop is requested.
Drop-in for ``time.sleep`` in the scrape loops, inter-outcode pauses and
retry backoffs so a pending Ctrl+C is never stuck behind a multi-second
delay."""
if seconds <= 0:
return
_STOP.wait(seconds)
def install_signal_handlers() -> None:
"""Route SIGINT/SIGTERM to :func:`request_stop` (second signal = hard exit).
Signal handlers run in the main thread, so this must be called from there."""
def _handle(signum, _frame):
if _STOP.is_set():
# Operator pressed Ctrl+C twice: abandon the graceful unwind.
raise KeyboardInterrupt
log.warning(
"Signal %s received; finishing in-flight work and saving partial "
"results (press Ctrl+C again to force-quit)...",
signal.Signals(signum).name,
)
_STOP.set()
for sig in (signal.SIGINT, signal.SIGTERM):
signal.signal(sig, _handle)

View file

@ -0,0 +1,90 @@
"""Tests for the shared request rate limiter (http_client.RateLimiter).
Uses a fake clock so the spacing logic is asserted deterministically rather than
by sleeping for real."""
import threading
import pytest
import http_client
import shutdown
from http_client import RateLimiter
class _FakeClock:
"""A monotonic clock that only advances when sleep() is called."""
def __init__(self, start=1000.0):
self.t = start
self.sleeps = []
def monotonic(self):
return self.t
def sleep(self, seconds):
self.sleeps.append(seconds)
self.t += seconds
def _install(monkeypatch):
clock = _FakeClock()
monkeypatch.setattr(http_client.time, "monotonic", clock.monotonic)
# The limiter waits via the interruptible shutdown.sleep so a Ctrl+C wakes a
# worker parked on a reserved rate-limit slot instead of stalling the exit.
monkeypatch.setattr(shutdown, "sleep", clock.sleep)
return clock
def test_first_acquire_does_not_sleep(monkeypatch):
clock = _install(monkeypatch)
RateLimiter(10).acquire()
assert clock.sleeps == []
def test_subsequent_acquires_are_spaced_by_the_interval(monkeypatch):
clock = _install(monkeypatch)
rl = RateLimiter(10) # 0.1s minimum interval
rl.acquire() # no wait
rl.acquire() # waits ~0.1
rl.acquire() # waits ~0.1
assert clock.sleeps == pytest.approx([0.1, 0.1])
def test_acquire_after_idle_gap_does_not_sleep(monkeypatch):
clock = _install(monkeypatch)
rl = RateLimiter(10)
rl.acquire()
clock.t += 5.0 # plenty of idle time passes
rl.acquire()
assert clock.sleeps == [] # the slot was already free
def test_zero_rate_disables_limiting(monkeypatch):
def _boom(_seconds): # pragma: no cover - must never be called
raise AssertionError("disabled limiter must not sleep")
monkeypatch.setattr(shutdown, "sleep", _boom)
rl = RateLimiter(0)
for _ in range(100):
rl.acquire()
def test_concurrent_acquires_are_all_spaced(monkeypatch):
# Real clock, tiny rate: N threads hitting acquire() at once must be
# serialised so the total wall time is at least (N-1) * interval.
rl = RateLimiter(200) # 5ms interval — fast but measurable
barrier = threading.Barrier(8)
def worker():
barrier.wait()
rl.acquire()
start = http_client.time.monotonic()
threads = [threading.Thread(target=worker) for _ in range(8)]
for t in threads:
t.start()
for t in threads:
t.join()
elapsed = http_client.time.monotonic() - start
assert elapsed >= 7 * 0.005 * 0.5 # allow scheduling slack, but not ~0

57
finder/test_main.py Normal file
View file

@ -0,0 +1,57 @@
"""Tests for the CLI argument parsing in main.py."""
import pytest
import main
# ---------------------------------------------------------------------------
# selected_sources — comma-separated --source values
# ---------------------------------------------------------------------------
def test_all_expands_to_every_source():
assert main.selected_sources("all") == ["rightmove", "onthemarket", "zoopla"]
def test_single_source():
assert main.selected_sources("rightmove") == ["rightmove"]
def test_comma_separated_sources():
assert main.selected_sources("rightmove,onthemarket") == [
"rightmove",
"onthemarket",
]
def test_whitespace_and_case_are_tolerated():
assert main.selected_sources(" Rightmove , ZOOPLA ") == ["rightmove", "zoopla"]
def test_order_is_canonical_and_deduplicated():
# Caller order ignored, duplicates collapsed: keeps SOURCES order so the
# downstream merge/dedup stays deterministic.
assert main.selected_sources("zoopla,rightmove,rightmove") == [
"rightmove",
"zoopla",
]
def test_all_anywhere_in_the_list_wins():
assert main.selected_sources("rightmove,all") == [
"rightmove",
"onthemarket",
"zoopla",
]
def test_empty_value_is_rejected():
with pytest.raises(SystemExit):
main.selected_sources(" , ")
def test_unknown_source_is_rejected():
with pytest.raises(SystemExit) as excinfo:
main.selected_sources("rightmove,foo")
assert "foo" in str(excinfo.value)

View file

@ -0,0 +1,58 @@
"""Tests for the persistent detail-fetch cache (postcode_cache.py)."""
from postcode_cache import load_cache, save_cache
def test_save_then_load_round_trips_str_and_none_values(tmp_path):
path = tmp_path / "rightmove.json"
data = {"123": "SW9 0HD", "456": None, "789": "E20 1FH"}
save_cache(path, data)
assert load_cache(path) == data
def test_save_then_load_round_trips_nested_dicts(tmp_path):
# Zoopla caches a flat dict of primitives (or None) per listing id.
path = tmp_path / "zoopla.json"
data = {
"59888978": {
"lat": 51.4,
"lng": -0.11,
"postcode": "BR1 2AB",
"uprn": "100023",
"number_or_name": "12",
"full_address": "12 High St",
},
"broken": None,
}
save_cache(path, data)
assert load_cache(path) == data
def test_load_missing_file_returns_empty(tmp_path):
assert load_cache(tmp_path / "does-not-exist.json") == {}
def test_load_corrupt_file_returns_empty(tmp_path):
path = tmp_path / "corrupt.json"
path.write_text("{not valid json", encoding="utf-8")
assert load_cache(path) == {}
def test_load_non_object_file_returns_empty(tmp_path):
path = tmp_path / "list.json"
path.write_text("[1, 2, 3]", encoding="utf-8")
assert load_cache(path) == {}
def test_save_creates_parent_directory(tmp_path):
path = tmp_path / "nested" / "dir" / "cache.json"
save_cache(path, {"a": "B1 1AA"})
assert path.exists()
assert load_cache(path) == {"a": "B1 1AA"}
def test_save_overwrites_existing(tmp_path):
path = tmp_path / "c.json"
save_cache(path, {"a": "X1 1AA"})
save_cache(path, {"b": "Y2 2BB"})
assert load_cache(path) == {"b": "Y2 2BB"}

View file

@ -0,0 +1,165 @@
"""Tests for Rightmove's accurate-pin skip and concurrent detail fetching."""
import threading
import rightmove
from rightmove import _needs_detail_fetch, _paginate, _prime_detail_postcodes
def _prop(pid, pin=None):
loc = {"latitude": 51.5, "longitude": -0.1}
if pin is not None:
loc["pinType"] = pin
return {"id": pid, "location": loc}
# ---------------------------------------------------------------------------
# _needs_detail_fetch — accurate-pin skip
# ---------------------------------------------------------------------------
def test_needs_detail_fetch_skips_only_accurate_pins(monkeypatch):
monkeypatch.setattr(rightmove, "RIGHTMOVE_SKIP_DETAILS_FOR_ACCURATE_PINS", True)
assert _needs_detail_fetch(_prop("1", "ACCURATE_POINT")) is False
assert _needs_detail_fetch(_prop("2", "APPROXIMATE_POINT")) is True
assert _needs_detail_fetch(_prop("3")) is True # no pinType -> fetch
assert _needs_detail_fetch({"id": "4"}) is True # no location -> fetch
def test_needs_detail_fetch_disabled_always_fetches(monkeypatch):
monkeypatch.setattr(rightmove, "RIGHTMOVE_SKIP_DETAILS_FOR_ACCURATE_PINS", False)
assert _needs_detail_fetch(_prop("1", "ACCURATE_POINT")) is True
# ---------------------------------------------------------------------------
# _prime_detail_postcodes — worklist selection + concurrent fetch
# ---------------------------------------------------------------------------
def _record_fetcher(monkeypatch):
fetched = []
lock = threading.Lock()
def fake_fetch(_client, pid):
with lock:
fetched.append(pid)
rightmove._detail_postcode_cache[pid] = f"PC-{pid}"
return f"PC-{pid}"
monkeypatch.setattr(rightmove, "_fetch_detail_postcode", fake_fetch)
return fetched
def test_prime_skips_cached_accurate_and_dedups(monkeypatch):
rightmove._detail_postcode_cache.clear()
rightmove._detail_postcode_cache["cached"] = "SW9 9ZZ"
monkeypatch.setattr(rightmove, "RIGHTMOVE_SKIP_DETAILS_FOR_ACCURATE_PINS", True)
fetched = _record_fetcher(monkeypatch)
props = [
_prop("cached", "APPROXIMATE_POINT"), # already cached -> skip
_prop("acc", "ACCURATE_POINT"), # accurate pin -> skip
_prop("a", "APPROXIMATE_POINT"), # fetch
_prop("a", "APPROXIMATE_POINT"), # duplicate id -> skip
_prop("b"), # no pinType -> fetch
{"id": ""}, # no id -> skip
]
_prime_detail_postcodes(object(), props, True, 10)
assert sorted(fetched) == ["a", "b"]
assert rightmove._detail_postcode_cache["a"] == "PC-a"
assert rightmove._detail_postcode_cache["b"] == "PC-b"
rightmove._detail_postcode_cache.clear()
def test_prime_respects_detail_cap(monkeypatch):
rightmove._detail_postcode_cache.clear()
monkeypatch.setattr(rightmove, "RIGHTMOVE_SKIP_DETAILS_FOR_ACCURATE_PINS", True)
fetched = _record_fetcher(monkeypatch)
props = [_prop(str(i), "APPROXIMATE_POINT") for i in range(10)]
_prime_detail_postcodes(object(), props, True, 3)
assert sorted(fetched) == ["0", "1", "2"]
rightmove._detail_postcode_cache.clear()
def test_prime_is_a_noop_when_disabled_or_cap_zero(monkeypatch):
rightmove._detail_postcode_cache.clear()
def _boom(*_a): # pragma: no cover - must never be called
raise AssertionError("must not fetch when disabled")
monkeypatch.setattr(rightmove, "_fetch_detail_postcode", _boom)
_prime_detail_postcodes(object(), [_prop("a")], False, 10) # fetch_details off
_prime_detail_postcodes(object(), [_prop("a")], True, 0) # cap 0
assert rightmove._detail_postcode_cache == {}
# ---------------------------------------------------------------------------
# _paginate — end-to-end (network stubbed): accurate pins fall back to
# coordinates, approximate pins use the detail postcode.
# ---------------------------------------------------------------------------
class _StubIndex:
def __init__(self, postcode):
self._postcode = postcode
def nearest(self, _lat, _lng):
return self._postcode
def _full_prop(pid, pin):
return {
"id": pid,
"location": {"latitude": 51.477, "longitude": -0.116, "pinType": pin},
"price": {"amount": 500000, "displayPrices": []},
"displayAddress": "Caldwell Street, Stockwell, London, SW9",
"propertySubType": "Flat",
"bedrooms": 2,
"bathrooms": 1,
"propertyUrl": f"/properties/{pid}",
}
def test_paginate_uses_detail_for_approximate_and_coords_for_accurate(monkeypatch):
rightmove._detail_postcode_cache.clear()
monkeypatch.setattr(rightmove, "RIGHTMOVE_SKIP_DETAILS_FOR_ACCURATE_PINS", True)
raw = [
_full_prop("approx", "APPROXIMATE_POINT"),
_full_prop("acc", "ACCURATE_POINT"),
]
monkeypatch.setattr(
rightmove, "_collect_search_props", lambda *a, **k: (raw, len(raw))
)
fetched = []
def fake_fetch(_client, pid):
fetched.append(pid)
rightmove._detail_postcode_cache[pid] = "SW9 0HD"
return "SW9 0HD"
monkeypatch.setattr(rightmove, "_fetch_detail_postcode", fake_fetch)
props, count = _paginate(
object(),
"1",
"SW9",
{"channel": "BUY", "transactionType": "BUY", "sortType": "2"},
_StubIndex("SW9 1AA"),
fetch_details=True,
detail_cap=10,
)
assert count == 2
assert fetched == ["approx"] # accurate pin was never fetched
by_id = {p["id"]: p for p in props}
assert by_id["approx"]["Postcode"] == "SW9 0HD"
assert by_id["approx"]["Postcode source"] == "detail_address"
# Accurate pin keeps the coordinate-nearest postcode.
assert by_id["acc"]["Postcode"] == "SW9 1AA"
assert by_id["acc"]["Postcode source"] != "detail_address"
rightmove._detail_postcode_cache.clear()

View file

@ -0,0 +1,150 @@
"""Tests for the orchestrator's provider parallelism and cache persistence."""
import threading
import onthemarket
import rightmove
import scraper
import zoopla
# ---------------------------------------------------------------------------
# _run_sources — Zoopla inline, others in threads, failures isolated
# ---------------------------------------------------------------------------
def test_run_sources_runs_every_source_and_isolates_failures():
order = []
lock = threading.Lock()
def rm():
with lock:
order.append("rm")
def otm():
with lock:
order.append("otm")
raise RuntimeError("otm boom")
def zoo():
with lock:
order.append("zoo")
errors: list[str] = []
scraper._run_sources(zoo, [("rightmove", rm), ("onthemarket", otm)], errors)
assert set(order) == {"rm", "otm", "zoo"}
# The failing source is recorded but did not stop the others.
assert any("onthemarket" in e and "boom" in e for e in errors)
assert not any("rightmove" in e for e in errors)
def test_run_sources_records_zoopla_failure():
errors: list[str] = []
def zoo():
raise ValueError("zoo down")
scraper._run_sources(zoo, [], errors)
assert any("zoopla" in e and "zoo down" in e for e in errors)
def test_run_sources_handles_no_background_runners():
ran = []
errors: list[str] = []
scraper._run_sources(lambda: ran.append("z"), [], errors)
assert ran == ["z"]
assert errors == []
def test_run_sources_handles_zoopla_absent():
ran = []
errors: list[str] = []
scraper._run_sources(None, [("rightmove", lambda: ran.append("rm"))], errors)
assert ran == ["rm"]
assert errors == []
# ---------------------------------------------------------------------------
# Persistent cache wiring
# ---------------------------------------------------------------------------
def test_seed_and_save_detail_caches_round_trip(tmp_path):
rightmove._detail_postcode_cache.clear()
rightmove._detail_postcode_cache.update({"111": "SW9 0HD", "222": None})
scraper._save_detail_caches(["rightmove"], tmp_path)
assert (tmp_path / "rightmove.json").exists()
rightmove._detail_postcode_cache.clear()
scraper._seed_detail_caches(["rightmove"], tmp_path)
assert rightmove._detail_postcode_cache == {"111": "SW9 0HD", "222": None}
rightmove._detail_postcode_cache.clear()
def test_seed_detail_caches_tolerates_missing_files(tmp_path):
rightmove._detail_postcode_cache.clear()
# No file written yet — seeding must not raise and must leave cache empty.
scraper._seed_detail_caches(["rightmove"], tmp_path)
assert rightmove._detail_postcode_cache == {}
# ---------------------------------------------------------------------------
# run_scrape — full orchestration wiring (sources stubbed, no network)
# ---------------------------------------------------------------------------
def _fake_prop(pid, postcode):
# Minimal shape: only the fields _merge_properties / _dedup_key read.
return {"id": pid, "Postcode": postcode, "price": 100000, "Bedrooms": 2}
def test_run_scrape_runs_all_sources_merges_and_persists_caches(tmp_path, monkeypatch):
for module in (rightmove, onthemarket, zoopla):
module.detail_cache_snapshot() # ensure hooks exist
rightmove._detail_postcode_cache.clear()
onthemarket._detail_postcode_cache.clear()
zoopla._detail_cache.clear()
def fake_rm(outcodes, pc_index, results, errors, cap):
results["rightmove"].append(_fake_prop("rm_1", "SW9 0HD"))
def fake_otm(outcodes, pc_index, results, errors, cap):
results["onthemarket"].append(_fake_prop("otm_1", "E1 6AN"))
def fake_zoo(outcodes, pc_index, pc_coords, results, errors, cap):
results["zoopla"].append(_fake_prop("z_1", "BR1 2AB"))
monkeypatch.setattr(scraper, "_scrape_rightmove", fake_rm)
monkeypatch.setattr(scraper, "_scrape_onthemarket", fake_otm)
monkeypatch.setattr(scraper, "_scrape_zoopla", fake_zoo)
monkeypatch.setattr(scraper, "build_postcode_coords", lambda: {})
written = {}
monkeypatch.setattr(
scraper,
"write_parquet",
lambda props, path: written.update(count=len(props), path=path),
)
result = scraper.run_scrape(
["SW9", "E1", "BR1"],
pc_index=None,
pc_coords=None,
sources=["rightmove", "onthemarket", "zoopla"],
output_dir=tmp_path,
max_properties_per_source=None,
)
# All three sources ran and their listings were merged + written.
assert written["count"] == 3
assert result["counts"]["total"] == 3
assert result["errors"] == []
# Persistent cache files were created for every selected source.
for source in ("rightmove", "onthemarket", "zoopla"):
assert (tmp_path / "detail_cache" / f"{source}.json").exists()
rightmove._detail_postcode_cache.clear()
onthemarket._detail_postcode_cache.clear()
zoopla._detail_cache.clear()

130
finder/test_shutdown.py Normal file
View file

@ -0,0 +1,130 @@
"""Tests for cooperative Ctrl+C shutdown across the scrapers."""
import signal
import threading
import time
import pytest
import http_client
import scraper
import shutdown
@pytest.fixture(autouse=True)
def _clean_stop_flag():
"""Every test starts (and leaves) the global stop flag cleared."""
shutdown.reset()
yield
shutdown.reset()
# ---------------------------------------------------------------------------
# The flag primitive
# ---------------------------------------------------------------------------
def test_request_and_reset_toggle_the_flag():
assert shutdown.stop_requested() is False
shutdown.request_stop()
assert shutdown.stop_requested() is True
shutdown.reset()
assert shutdown.stop_requested() is False
def test_sleep_returns_immediately_once_stop_requested():
shutdown.request_stop()
started = time.monotonic()
shutdown.sleep(30) # would hang for 30s without the early wake
assert time.monotonic() - started < 1.0
def test_sleep_wakes_when_stop_requested_from_another_thread():
def _trip():
time.sleep(0.05)
shutdown.request_stop()
waker = threading.Thread(target=_trip)
waker.start()
started = time.monotonic()
shutdown.sleep(30)
waker.join()
assert time.monotonic() - started < 1.0
# ---------------------------------------------------------------------------
# Signal handlers: first signal requests stop, second forces a hard exit
# ---------------------------------------------------------------------------
def test_signal_handler_requests_stop_then_escalates():
previous = signal.getsignal(signal.SIGINT)
try:
shutdown.install_signal_handlers()
handler = signal.getsignal(signal.SIGINT)
assert callable(handler)
# First Ctrl+C: graceful stop requested, no exception.
handler(signal.SIGINT, None)
assert shutdown.stop_requested() is True
# Second Ctrl+C: escalate to a hard interrupt.
with pytest.raises(KeyboardInterrupt):
handler(signal.SIGINT, None)
finally:
signal.signal(signal.SIGINT, previous)
# ---------------------------------------------------------------------------
# HTTP layer bails out instead of issuing more requests
# ---------------------------------------------------------------------------
def test_fetch_with_retry_returns_none_without_touching_client():
class _ExplodingClient:
def get(self, *_a, **_k): # pragma: no cover - must never be called
raise AssertionError("no request should be made after shutdown")
shutdown.request_stop()
assert http_client.fetch_with_retry(_ExplodingClient(), "http://x") is None
# ---------------------------------------------------------------------------
# A source loop stops at the next outcode boundary
# ---------------------------------------------------------------------------
def _londonish_prop(outcode):
return {"id": outcode, "Postcode": f"{outcode} 0HD", "price": 100000, "Bedrooms": 2}
def test_scrape_rightmove_stops_after_signal(monkeypatch):
visited = []
class _DummyClient:
def close(self):
pass
monkeypatch.setattr(scraper, "make_client", lambda: _DummyClient())
monkeypatch.setattr(scraper, "resolve_outcode_id", lambda _client, oc: oc)
monkeypatch.setattr(scraper, "DELAY_BETWEEN_OUTCODES", 0)
def fake_search(_client, _oid, outcode, *_a, **_k):
visited.append(outcode)
# Operator hits Ctrl+C while the first outcode is being processed.
shutdown.request_stop()
return [_londonish_prop(outcode)]
monkeypatch.setattr(scraper, "rightmove_search_outcode", fake_search)
results = {source: [] for source in scraper.SOURCE_ORDER}
errors: list[str] = []
scraper._scrape_rightmove(
["SW9", "E1", "BR1"], pc_index=None, results=results,
errors=errors, max_properties_per_source=None,
)
# Only the first outcode ran; the loop bailed before the next boundary.
assert visited == ["SW9"]
assert len(results["rightmove"]) == 1
assert errors == []

View file

@ -29,6 +29,7 @@ from urllib.parse import parse_qsl, urlencode, urljoin, urlparse, urlunparse
import httpx
import shutdown
from constants import (
DATA_DIR,
DELAY_BETWEEN_PAGES,
@ -973,7 +974,7 @@ def _paginate(
listing["_detail"] = fetch_detail(url)
if not cached:
detail_state["fetched"] += 1
time.sleep(DELAY_BETWEEN_PAGES)
shutdown.sleep(DELAY_BETWEEN_PAGES)
all_listings = _extract_listings(page)
for listing in all_listings:
@ -988,13 +989,15 @@ def _paginate(
page_num = 2
while True:
if shutdown.stop_requested():
break
next_url = _find_next_page_url(page)
if not next_url:
if total_results > 0 and len(all_listings) >= total_results:
break
next_url = _url_with_page(page.url, page_num)
time.sleep(DELAY_BETWEEN_PAGES)
shutdown.sleep(DELAY_BETWEEN_PAGES)
try:
page.goto(next_url, wait_until="domcontentloaded", timeout=30000)
@ -1119,9 +1122,23 @@ def _extract_outcode(text: str) -> str | None:
# listingId -> parsed detail dict (or None). Failures are cached too, so a
# broken listing is not re-fetched within a run (the same listing reappears
# across overlapping outcode searches).
# across overlapping outcode searches). Seeded from / dumped to a persistent
# on-disk cache by the orchestrator (see postcode_cache.py) so a recurring
# scrape only re-fetches newly-listed properties — the biggest saving for
# Zoopla, whose detail fetch drives a real browser tab.
_detail_cache: dict[str, dict | None] = {}
def seed_detail_cache(mapping: dict) -> None:
"""Pre-populate the in-memory detail cache from a persisted snapshot."""
if mapping:
_detail_cache.update(mapping)
def detail_cache_snapshot() -> dict:
"""Return a JSON-serialisable copy of the in-memory detail cache."""
return dict(_detail_cache)
_LISTING_ID_RE = re.compile(r"/details/(\d+)/?")
# The property's own location is carried by a `"location":{...}` wrapper and a

View file

@ -92,6 +92,7 @@ interface FiltersProps {
onTravelTimeDragEnd: (index: number) => void;
onTravelTimeToggleBest: (index: number) => void;
onTravelTimeToggleNoChange: (index: number) => void;
onTravelTimeToggleOneChange: (index: number) => void;
onTravelTimeToggleNoBuses: (index: number) => void;
aiFilterLoading: boolean;
aiFilterError: string | null;
@ -139,6 +140,7 @@ export default memo(function Filters({
onTravelTimeDragEnd,
onTravelTimeToggleBest,
onTravelTimeToggleNoChange,
onTravelTimeToggleOneChange,
onTravelTimeToggleNoBuses,
aiFilterLoading,
aiFilterError,
@ -667,6 +669,7 @@ export default memo(function Filters({
onTravelTimeDragEnd={onTravelTimeDragEnd}
onTravelTimeToggleBest={onTravelTimeToggleBest}
onTravelTimeToggleNoChange={onTravelTimeToggleNoChange}
onTravelTimeToggleOneChange={onTravelTimeToggleOneChange}
onTravelTimeToggleNoBuses={onTravelTimeToggleNoBuses}
/>

View file

@ -181,6 +181,7 @@ export default function MapPage({
handleTimeRangeChange,
handleToggleBest,
handleToggleNoChange,
handleToggleOneChange,
handleToggleNoBuses,
} = useTravelTime(initialTravelTime);
@ -248,6 +249,7 @@ export default function MapPage({
representable.map(({ tt, parsed }) => ({
mode: parsed.mode,
noChange: parsed.noChange,
oneChange: parsed.oneChange,
noBuses: parsed.noBuses,
slug: tt.slug,
label: tt.label,
@ -800,6 +802,7 @@ export default function MapPage({
onTravelTimeDragEnd={handleTravelTimeDragEnd}
onTravelTimeToggleBest={handleToggleBest}
onTravelTimeToggleNoChange={handleToggleNoChange}
onTravelTimeToggleOneChange={handleToggleOneChange}
onTravelTimeToggleNoBuses={handleToggleNoBuses}
aiFilterLoading={aiFilterLoading}
aiFilterError={aiFilterError}
@ -853,6 +856,7 @@ export default function MapPage({
handleToggleBest,
handleToggleNoBuses,
handleToggleNoChange,
handleToggleOneChange,
handleTogglePin,
handleTravelTimeDragEnd,
handleTravelTimeRemoveEntry,

View file

@ -27,6 +27,7 @@ interface TravelTimeCardProps {
timeRange: [number, number] | null;
useBest: boolean;
noChange: boolean;
oneChange: boolean;
noBuses: boolean;
isPinned: boolean;
isActive: boolean;
@ -39,6 +40,7 @@ interface TravelTimeCardProps {
onDragEnd: () => void;
onToggleBest: () => void;
onToggleNoChange: () => void;
onToggleOneChange: () => void;
onToggleNoBuses: () => void;
onRemove: () => void;
filterImpact?: number;
@ -52,6 +54,7 @@ export function TravelTimeCard({
timeRange,
useBest,
noChange,
oneChange,
noBuses,
isPinned,
isActive,
@ -64,6 +67,7 @@ export function TravelTimeCard({
onDragEnd,
onToggleBest,
onToggleNoChange,
onToggleOneChange,
onToggleNoBuses,
onRemove,
filterImpact,
@ -75,6 +79,7 @@ export function TravelTimeCard({
const [showInfo, setShowInfo] = useState(false);
const [showBestInfo, setShowBestInfo] = useState(false);
const [showNoChangeInfo, setShowNoChangeInfo] = useState(false);
const [showOneChangeInfo, setShowOneChangeInfo] = useState(false);
const [showNoBusesInfo, setShowNoBusesInfo] = useState(false);
const handleDestinationSelect = useCallback(
@ -169,6 +174,20 @@ export function TravelTimeCard({
<InfoIcon className="w-3 h-3" />
</IconButton>
</div>
<div className="flex items-center gap-0.5">
<PillToggle
label={t('travel.oneChange')}
active={oneChange}
onClick={onToggleOneChange}
size="xs"
/>
<IconButton
onClick={() => setShowOneChangeInfo(true)}
title={t('travel.oneChangeTitle')}
>
<InfoIcon className="w-3 h-3" />
</IconButton>
</div>
<div className="flex items-center gap-0.5">
<PillToggle
label={t('travel.noBuses')}
@ -201,6 +220,14 @@ export function TravelTimeCard({
</InfoPopup>
)}
{showOneChangeInfo && (
<InfoPopup title={t('travel.oneChangeTitle')} onClose={() => setShowOneChangeInfo(false)}>
<p className="text-sm text-warm-700 dark:text-warm-300 leading-relaxed">
<Trans i18nKey="travel.oneChangeDesc" components={{ strong: <strong /> }} />
</p>
</InfoPopup>
)}
{showNoBusesInfo && (
<InfoPopup title={t('travel.noBusesTitle')} onClose={() => setShowNoBusesInfo(false)}>
<p className="text-sm text-warm-700 dark:text-warm-300 leading-relaxed">

View file

@ -59,6 +59,7 @@ interface ActiveFilterListProps {
onTravelTimeDragEnd: (index: number) => void;
onTravelTimeToggleBest: (index: number) => void;
onTravelTimeToggleNoChange: (index: number) => void;
onTravelTimeToggleOneChange: (index: number) => void;
onTravelTimeToggleNoBuses: (index: number) => void;
}
@ -88,6 +89,7 @@ export function ActiveFilterList({
onTravelTimeDragEnd,
onTravelTimeToggleBest,
onTravelTimeToggleNoChange,
onTravelTimeToggleOneChange,
onTravelTimeToggleNoBuses,
}: ActiveFilterListProps) {
const travelCards = (
@ -105,6 +107,7 @@ export function ActiveFilterList({
onTravelTimeDragEnd={onTravelTimeDragEnd}
onTravelTimeToggleBest={onTravelTimeToggleBest}
onTravelTimeToggleNoChange={onTravelTimeToggleNoChange}
onTravelTimeToggleOneChange={onTravelTimeToggleOneChange}
onTravelTimeToggleNoBuses={onTravelTimeToggleNoBuses}
onDragStart={onDragStart}
onDragChange={onDragChange}

View file

@ -57,6 +57,7 @@ interface ActiveFiltersPanelProps {
onTravelTimeDragEnd: (index: number) => void;
onTravelTimeToggleBest: (index: number) => void;
onTravelTimeToggleNoChange: (index: number) => void;
onTravelTimeToggleOneChange: (index: number) => void;
onTravelTimeToggleNoBuses: (index: number) => void;
}
@ -102,6 +103,7 @@ export function ActiveFiltersPanel({
onTravelTimeDragEnd,
onTravelTimeToggleBest,
onTravelTimeToggleNoChange,
onTravelTimeToggleOneChange,
onTravelTimeToggleNoBuses,
}: ActiveFiltersPanelProps) {
const { t } = useTranslation();
@ -206,6 +208,7 @@ export function ActiveFiltersPanel({
onTravelTimeDragEnd={onTravelTimeDragEnd}
onTravelTimeToggleBest={onTravelTimeToggleBest}
onTravelTimeToggleNoChange={onTravelTimeToggleNoChange}
onTravelTimeToggleOneChange={onTravelTimeToggleOneChange}
onTravelTimeToggleNoBuses={onTravelTimeToggleNoBuses}
/>
</div>

View file

@ -21,6 +21,7 @@ interface TravelTimeFilterCardsProps {
onTravelTimeDragEnd: (index: number) => void;
onTravelTimeToggleBest: (index: number) => void;
onTravelTimeToggleNoChange: (index: number) => void;
onTravelTimeToggleOneChange: (index: number) => void;
onTravelTimeToggleNoBuses: (index: number) => void;
onDragStart: (name: string, initialValue?: [number, number]) => void;
onDragChange: (value: [number, number]) => void;
@ -40,6 +41,7 @@ export function TravelTimeFilterCards({
onTravelTimeDragEnd,
onTravelTimeToggleBest,
onTravelTimeToggleNoChange,
onTravelTimeToggleOneChange,
onTravelTimeToggleNoBuses,
onDragStart,
onDragChange,
@ -57,6 +59,7 @@ export function TravelTimeFilterCards({
timeRange={entry.timeRange}
useBest={entry.useBest}
noChange={entry.noChange ?? false}
oneChange={entry.oneChange ?? false}
noBuses={entry.noBuses ?? false}
isPinned={pinnedFeature === fieldKey}
isActive={activeFeature === fieldKey}
@ -71,6 +74,7 @@ export function TravelTimeFilterCards({
onDragEnd={() => onTravelTimeDragEnd(index)}
onToggleBest={() => onTravelTimeToggleBest(index)}
onToggleNoChange={() => onTravelTimeToggleNoChange(index)}
onToggleOneChange={() => onTravelTimeToggleOneChange(index)}
onToggleNoBuses={() => onTravelTimeToggleNoBuses(index)}
onRemove={() => onTravelTimeRemoveEntry(index)}
filterImpact={filterImpacts?.[fieldKey]}

View file

@ -150,6 +150,12 @@ export function useExportController({
const params = new URLSearchParams();
if (isListMode) {
params.set('postcodes', postcodeList.join(','));
// Pass the active filters so the export keeps its two sheets — "Selected"
// (the filtered feature columns) and "All Data" — even in list mode. The
// server uses them only to pick columns; the supplied postcodes still
// drive which rows appear.
const filterStr = buildFilterString(filters, features);
if (filterStr) params.set('filters', filterStr);
if (shareCode) params.set('share', shareCode);
} else {
const { south, west, north, east } = bounds!;

View file

@ -34,6 +34,7 @@ describe('useTravelTime', () => {
timeRange: null,
useBest: false,
noChange: false,
oneChange: false,
noBuses: false,
},
]);
@ -143,6 +144,27 @@ describe('useTravelTime', () => {
act(() => result.current.handleToggleNoChange(0));
expect(result.current.entries[0]).toMatchObject({ noChange: false, noBuses: true });
});
it('keeps noChange and oneChange mutually exclusive', () => {
const { result } = renderHook(() => useTravelTime());
act(() => result.current.handleAddEntry('transit'));
act(() => result.current.handleSetDestination(0, 'bank', 'Bank'));
act(() => result.current.handleToggleNoChange(0));
expect(result.current.entries[0]).toMatchObject({ noChange: true, oneChange: false });
// Enabling oneChange clears noChange...
act(() => result.current.handleToggleOneChange(0));
expect(result.current.entries[0]).toMatchObject({ noChange: false, oneChange: true });
// ...and vice versa.
act(() => result.current.handleToggleNoChange(0));
expect(result.current.entries[0]).toMatchObject({ noChange: true, oneChange: false });
// Both can be off (no restriction): toggling the active one off leaves neither set.
act(() => result.current.handleToggleNoChange(0));
expect(result.current.entries[0]).toMatchObject({ noChange: false, oneChange: false });
});
});
describe('resolveTransitVariant', () => {
@ -163,10 +185,20 @@ describe('resolveTransitVariant', () => {
it('maps transit toggle combinations to the right variant string', () => {
expect(resolveTransitVariant(base)).toBe('transit');
expect(resolveTransitVariant({ ...base, noChange: true })).toBe('transit-no-change');
expect(resolveTransitVariant({ ...base, oneChange: true })).toBe('transit-one-change');
expect(resolveTransitVariant({ ...base, noBuses: true })).toBe('transit-no-bus');
expect(resolveTransitVariant({ ...base, noChange: true, noBuses: true })).toBe(
'transit-no-change-no-bus'
);
expect(resolveTransitVariant({ ...base, oneChange: true, noBuses: true })).toBe(
'transit-one-change-no-bus'
);
});
it('lets noChange win if both change flags are somehow set', () => {
expect(resolveTransitVariant({ ...base, noChange: true, oneChange: true })).toBe(
'transit-no-change'
);
});
it('treats undefined flags as false', () => {
@ -177,47 +209,69 @@ describe('resolveTransitVariant', () => {
});
describe('parseServerMode', () => {
it('round-trips the four toggle-reachable variants', () => {
it('round-trips the six toggle-reachable transit variants', () => {
expect(parseServerMode('transit')).toEqual({
mode: 'transit',
noChange: false,
oneChange: false,
noBuses: false,
});
expect(parseServerMode('transit-no-bus')).toEqual({
mode: 'transit',
noChange: false,
oneChange: false,
noBuses: true,
});
expect(parseServerMode('transit-no-change')).toEqual({
mode: 'transit',
noChange: true,
oneChange: false,
noBuses: false,
});
expect(parseServerMode('transit-no-change-no-bus')).toEqual({
mode: 'transit',
noChange: true,
oneChange: false,
noBuses: true,
});
expect(parseServerMode('transit-one-change')).toEqual({
mode: 'transit',
noChange: false,
oneChange: true,
noBuses: false,
});
expect(parseServerMode('transit-one-change-no-bus')).toEqual({
mode: 'transit',
noChange: false,
oneChange: true,
noBuses: true,
});
});
it('parses non-transit base modes', () => {
expect(parseServerMode('car')).toEqual({ mode: 'car', noChange: false, noBuses: false });
expect(parseServerMode('car')).toEqual({
mode: 'car',
noChange: false,
oneChange: false,
noBuses: false,
});
expect(parseServerMode('bicycle')).toEqual({
mode: 'bicycle',
noChange: false,
oneChange: false,
noBuses: false,
});
expect(parseServerMode('walking')).toEqual({
mode: 'walking',
noChange: false,
oneChange: false,
noBuses: false,
});
});
it('returns null for variants the UI cannot represent (no silent broadening)', () => {
expect(parseServerMode('transit-one-change')).toBeNull();
expect(parseServerMode('transit-one-change-no-bus')).toBeNull();
expect(parseServerMode('unknown-mode')).toBeNull();
expect(parseServerMode('transit-two-change')).toBeNull();
});
it('travelFieldKey uses the resolved variant', () => {

View file

@ -66,6 +66,9 @@ export interface TravelTimeEntry {
useBest: boolean;
/** Restrict transit to walk-transit-walk (0 changes). Optional; defaults to false. */
noChange?: boolean;
/** Restrict transit to at most one change. Optional; defaults to false.
* Mutually exclusive with noChange (the UI never sets both). */
oneChange?: boolean;
/** Drop buses from the allowed transit modes. Optional; defaults to false. */
noBuses?: boolean;
}
@ -77,42 +80,44 @@ export interface TravelTimeEntry {
*
* For non-transit modes the entry.mode passes through unchanged.
*
* Note: the transit-one-change* variants exist server-side but are not reachable
* from the UI toggles (only no-change + no-buses are exposed). They're available
* via direct API access for callers that want them.
* noChange and oneChange are mutually exclusive (the UI never sets both); if
* they ever were, noChange wins here.
*/
export function resolveTransitVariant(entry: TravelTimeEntry): string {
if (entry.mode !== 'transit') return entry.mode;
const nc = entry.noChange ?? false;
const oc = entry.oneChange ?? false;
const nb = entry.noBuses ?? false;
if (nc && nb) return 'transit-no-change-no-bus';
if (nc) return 'transit-no-change';
if (nb) return 'transit-no-bus';
return 'transit';
if (nc) return nb ? 'transit-no-change-no-bus' : 'transit-no-change';
if (oc) return nb ? 'transit-one-change-no-bus' : 'transit-one-change';
return nb ? 'transit-no-bus' : 'transit';
}
/**
* Parse a server-side mode string (incl. transit variants) back into a base
* TransportMode + UI toggle booleans. Returns null for mode strings the UI
* cannot represent (currently: transit-one-change, transit-one-change-no-bus).
* Callers should skip entries that parse to null rather than silently
* normalising to a different variant.
* cannot represent. Callers should skip entries that parse to null rather than
* silently normalising to a different variant.
*/
export function parseServerMode(
modeStr: string
): { mode: TransportMode; noChange: boolean; noBuses: boolean } | null {
): { mode: TransportMode; noChange: boolean; oneChange: boolean; noBuses: boolean } | null {
if (modeStr === 'car' || modeStr === 'bicycle' || modeStr === 'walking') {
return { mode: modeStr, noChange: false, noBuses: false };
return { mode: modeStr, noChange: false, oneChange: false, noBuses: false };
}
switch (modeStr) {
case 'transit':
return { mode: 'transit', noChange: false, noBuses: false };
return { mode: 'transit', noChange: false, oneChange: false, noBuses: false };
case 'transit-no-bus':
return { mode: 'transit', noChange: false, noBuses: true };
return { mode: 'transit', noChange: false, oneChange: false, noBuses: true };
case 'transit-no-change':
return { mode: 'transit', noChange: true, noBuses: false };
return { mode: 'transit', noChange: true, oneChange: false, noBuses: false };
case 'transit-no-change-no-bus':
return { mode: 'transit', noChange: true, noBuses: true };
return { mode: 'transit', noChange: true, oneChange: false, noBuses: true };
case 'transit-one-change':
return { mode: 'transit', noChange: false, oneChange: true, noBuses: false };
case 'transit-one-change-no-bus':
return { mode: 'transit', noChange: false, oneChange: true, noBuses: true };
default:
return null;
}
@ -145,6 +150,7 @@ export function useTravelTime(initial?: TravelTimeInitial) {
timeRange: null,
useBest: false,
noChange: false,
oneChange: false,
noBuses: false,
},
]);
@ -182,11 +188,25 @@ export function useTravelTime(initial?: TravelTimeInitial) {
);
}, []);
// noChange and oneChange are mutually exclusive: enabling one clears the
// other so the resolved server variant is never ambiguous.
const handleToggleNoChange = useCallback((index: number) => {
setEntries((prev) =>
dedupeTravelTimeEntries(
prev.map((entry, i) =>
i === index ? { ...entry, noChange: !(entry.noChange ?? false) } : entry
i === index ? { ...entry, noChange: !(entry.noChange ?? false), oneChange: false } : entry
)
)
);
}, []);
const handleToggleOneChange = useCallback((index: number) => {
setEntries((prev) =>
dedupeTravelTimeEntries(
prev.map((entry, i) =>
i === index
? { ...entry, oneChange: !(entry.oneChange ?? false), noChange: false }
: entry
)
)
);
@ -219,6 +239,7 @@ export function useTravelTime(initial?: TravelTimeInitial) {
handleTimeRangeChange,
handleToggleBest,
handleToggleNoChange,
handleToggleOneChange,
handleToggleNoBuses,
};
}

View file

@ -96,19 +96,19 @@ export const details: Record<string, Record<string, string>> = {
'Median age':
"Provient du Census 2021 (TS007A). Âge médian des résidents habituels dans le LSOA, calculé par interpolation linéaire à partir des effectifs par tranche d'âge de cinq ans. Les zones à population plus jeune ont tendance à être urbaines, universitaires ou à accueillir davantage de familles ; les médianes plus élevées sont typiques des zones rurales et côtières.",
'% White':
"Provient du Census 2021. Pourcentage de la population de l'autorité locale s'identifiant comme Blanc (anglais, gallois, écossais, nord-irlandais, britannique, irlandais, Gitan ou Voyageur irlandais, Rom, ou tout autre origine blanche).",
"Provient du Census 2021. Pourcentage de la population du quartier (LSOA) s'identifiant comme Blanc (anglais, gallois, écossais, nord-irlandais, britannique, irlandais, Gitan ou Voyageur irlandais, Rom, ou tout autre origine blanche).",
'% South Asian':
"Provient du Census 2021. Pourcentage de la population de l'autorité locale s'identifiant comme Indien, Pakistanais, Bangladais ou toute autre origine asiatique.",
"Provient du Census 2021. Pourcentage de la population du quartier (LSOA) s'identifiant comme Indien, Pakistanais ou Bangladais.",
'% Black':
"Provient du Census 2021. Pourcentage de la population de l'autorité locale s'identifiant comme Noir, Noir britannique, Caribéen ou Africain.",
"Provient du Census 2021. Pourcentage de la population du quartier (LSOA) s'identifiant comme Noir, Noir britannique, Caribéen ou Africain.",
'% East Asian':
"Provient du Census 2021. Pourcentage de la population de l'autorité locale s'identifiant comme Chinois.",
"Provient du Census 2021. Pourcentage de la population du quartier (LSOA) s'identifiant comme Chinois.",
'% SE Asian':
"Provient du Census 2021. Pourcentage de la population de l'autorité locale s'identifiant comme appartenant à une autre origine est/sud-est asiatique (par ex. Philippin, Vietnamien ou Thaïlandais).",
"Provient du Census 2021. Pourcentage de la population du quartier (LSOA) s'identifiant comme appartenant à une autre origine est/sud-est asiatique (par ex. Philippin, Vietnamien ou Thaïlandais).",
'% Mixed':
"Provient du Census 2021. Pourcentage de la population de l'autorité locale s'identifiant comme Mixte ou appartenant à plusieurs groupes ethniques (Blanc et Noir caribéen, Blanc et Noir africain, Blanc et Asiatique, ou tout autre fond mixte ou multiple).",
"Provient du Census 2021. Pourcentage de la population du quartier (LSOA) s'identifiant comme Mixte ou appartenant à plusieurs groupes ethniques (Blanc et Noir caribéen, Blanc et Noir africain, Blanc et Asiatique, ou tout autre fond mixte ou multiple).",
'% Other':
"Provient du Census 2021. Pourcentage de la population de l'autorité locale s'identifiant comme appartenant à un autre groupe ethnique (Arabe ou tout autre groupe ethnique non couvert par les catégories principales).",
"Provient du Census 2021. Pourcentage de la population du quartier (LSOA) s'identifiant comme appartenant à un autre groupe ethnique (Arabe ou tout autre groupe ethnique non couvert par les catégories principales).",
'Voter turnout (%)':
"La proportion de l'électorat inscrit qui a voté de manière valide lors des élections générales britanniques de juillet 2024. Calculée comme le nombre de votes valides divisé par la taille de l'électorat. Une participation plus élevée est généralement corrélée avec des zones plus aisées et des scrutins plus serrés.",
'% Labour':
@ -236,19 +236,19 @@ export const details: Record<string, Record<string, string>> = {
'Median age':
'Aus dem Census 2021 (TS007A). Medianalter der ortsansässigen Bevölkerung im LSOA, berechnet durch lineare Interpolation aus Fünfjahres-Altersband-Zählungen. Gebiete mit jüngerer Bevölkerung sind tendenziell städtisch, Universitätsstädte oder haben mehr Familien; höhere Mediane sind typisch für ländliche und Küstengebiete.',
'% White':
'Aus dem Census 2021. Prozentsatz der Bevölkerung der Gemeinde, die sich als Weiß identifiziert (Englisch, Walisisch, Schottisch, Nordirisch, Britisch, Irisch, Sinti und Roma, Roma oder sonstiger weißer Hintergrund).',
'Aus dem Census 2021. Prozentsatz der Bevölkerung der Nachbarschaft (LSOA), die sich als Weiß identifiziert (Englisch, Walisisch, Schottisch, Nordirisch, Britisch, Irisch, Sinti und Roma, Roma oder sonstiger weißer Hintergrund).',
'% South Asian':
'Aus dem Census 2021. Prozentsatz der Bevölkerung der Gemeinde, die sich als Indisch, Pakistanisch, Bangladeschisch oder mit sonstigem asiatischen Hintergrund identifiziert.',
'Aus dem Census 2021. Prozentsatz der Bevölkerung der Nachbarschaft (LSOA), die sich als Indisch, Pakistanisch oder Bangladeschisch identifiziert.',
'% Black':
'Aus dem Census 2021. Prozentsatz der Bevölkerung der Gemeinde, die sich als Schwarz, Schwarz-Britisch, Karibisch oder Afrikanisch identifiziert.',
'Aus dem Census 2021. Prozentsatz der Bevölkerung der Nachbarschaft (LSOA), die sich als Schwarz, Schwarz-Britisch, Karibisch oder Afrikanisch identifiziert.',
'% East Asian':
'Aus dem Census 2021. Prozentsatz der Bevölkerung der Gemeinde, die sich als Chinesisch identifiziert.',
'Aus dem Census 2021. Prozentsatz der Bevölkerung der Nachbarschaft (LSOA), die sich als Chinesisch identifiziert.',
'% SE Asian':
'Aus dem Census 2021. Prozentsatz der Bevölkerung der Gemeinde, die sich mit einer anderen (nicht chinesischen) ost-/südostasiatischen Herkunft identifiziert, z. B. philippinisch, vietnamesisch oder thailändisch.',
'Aus dem Census 2021. Prozentsatz der Bevölkerung der Nachbarschaft (LSOA), die sich mit einer anderen (nicht chinesischen) ost-/südostasiatischen Herkunft identifiziert, z. B. philippinisch, vietnamesisch oder thailändisch.',
'% Mixed':
'Aus dem Census 2021. Prozentsatz der Bevölkerung der Gemeinde, die sich als gemischt oder mit mehreren ethnischen Zugehörigkeiten identifiziert (Weiß und Schwarzkaribisch, Weiß und Schwarzafrikanisch, Weiß und Asiatisch oder sonstiger gemischter Hintergrund).',
'Aus dem Census 2021. Prozentsatz der Bevölkerung der Nachbarschaft (LSOA), die sich als gemischt oder mit mehreren ethnischen Zugehörigkeiten identifiziert (Weiß und Schwarzkaribisch, Weiß und Schwarzafrikanisch, Weiß und Asiatisch oder sonstiger gemischter Hintergrund).',
'% Other':
'Aus dem Census 2021. Prozentsatz der Bevölkerung der Gemeinde, die sich als einer anderen ethnischen Gruppe zugehörig identifiziert (Arabisch oder eine andere ethnische Gruppe, die nicht von den Hauptkategorien abgedeckt wird).',
'Aus dem Census 2021. Prozentsatz der Bevölkerung der Nachbarschaft (LSOA), die sich als einer anderen ethnischen Gruppe zugehörig identifiziert (Arabisch oder eine andere ethnische Gruppe, die nicht von den Hauptkategorien abgedeckt wird).',
'Voter turnout (%)':
'Der Anteil der registrierten Wahlberechtigten, die bei der britischen Parlamentswahl im Juli 2024 eine gültige Stimme abgegeben haben. Berechnet als gültige Stimmen geteilt durch die Größe der Wählerschaft. Eine höhere Wahlbeteiligung korreliert im Allgemeinen mit wohlhabenderen Gebieten und knapperen Ergebnissen.',
'% Labour':
@ -374,17 +374,18 @@ export const details: Record<string, Record<string, string>> = {
'Median age':
'来自2021年CensusTS007A。通过对五岁年龄段人口数进行线性插值计算得出的LSOA常住居民年龄中位数。年轻人口集中的地区往往是城市、大学城或家庭聚居地年龄中位数较高的地区多见于农村和沿海地区。',
'% White':
'来自2021年Census。地方政府人口中认同为白人(英格兰人、威尔士人、苏格兰人、北爱尔兰人、英国人、爱尔兰人、吉普赛人或爱尔兰旅行者、罗姆人或其他白人背景)的百分比。',
'来自2021年Census。本地社区LSOA人口中认同为白人(英格兰人、威尔士人、苏格兰人、北爱尔兰人、英国人、爱尔兰人、吉普赛人或爱尔兰旅行者、罗姆人或其他白人背景)的百分比。',
'% South Asian':
'来自2021年Census。地方政府人口中认同为印度人、巴基斯坦人、孟加拉国人或其他亚洲背景的百分比。',
'% Black': '来自2021年Census。地方政府人口中认同为黑人、英国黑人、加勒比人或非洲人的百分比。',
'% East Asian': '来自2021年Census。地方政府人口中认同为华人的百分比。',
'来自2021年Census。本地社区LSOA人口中认同为印度人、巴基斯坦人或孟加拉国人的百分比。',
'% Black':
'来自2021年Census。本地社区LSOA人口中认同为黑人、英国黑人、加勒比人或非洲人的百分比。',
'% East Asian': '来自2021年Census。本地社区LSOA人口中认同为华人的百分比。',
'% SE Asian':
'来自2021年Census。地方政府人口中认同为其他(非华人)东亚/东南亚裔(如菲律宾裔、越南裔或泰裔)的百分比。',
'来自2021年Census。本地社区LSOA人口中认同为其他(非华人)东亚/东南亚裔(如菲律宾裔、越南裔或泰裔)的百分比。',
'% Mixed':
'来自2021年 Census。地方政府人口中认同为混血或多种族群体(白人与黑人加勒比裔、白人与黑人非洲裔、白人与亚洲裔,或其他混血或多种族背景)的百分比。',
'来自2021年 Census。本地社区LSOA人口中认同为混血或多种族群体(白人与黑人加勒比裔、白人与黑人非洲裔、白人与亚洲裔,或其他混血或多种族背景)的百分比。',
'% Other':
'来自2021年Census。地方政府人口中认同为其他族裔群体(阿拉伯人或其他未被主要类别涵盖的族裔)的百分比。',
'来自2021年Census。本地社区LSOA人口中认同为其他族裔群体(阿拉伯人或其他未被主要类别涵盖的族裔)的百分比。',
'Voter turnout (%)':
'2024年7月英国大选中投出有效选票的登记选民比例。计算方式为有效票数除以选民总数。较高的投票率通常与较富裕地区和竞争更激烈的选举相关。',
'% Labour':
@ -508,19 +509,19 @@ export const details: Record<string, Record<string, string>> = {
'Median age':
'Census 2021 (TS007A) से. LSOA में सामान्य निवासियों की मध्य आयु, पांच-वर्षीय आयु समूहों की गणना से रैखिक इंटरपोलेशन द्वारा निकाली गई. युवा आबादी वाले क्षेत्र आमतौर पर शहरी, विश्वविद्यालय नगर या अधिक परिवारों वाले होते हैं; अधिक मध्य आयु वाले क्षेत्र आमतौर पर ग्रामीण और तटीय इलाकों में मिलते हैं.',
'% White':
'Census 2021 से. स्थानीय प्राधिकरण की आबादी का प्रतिशत जो खुद को श्वेत (अंग्रेज़, वेल्श, स्कॉटिश, उत्तरी आयरिश, ब्रिटिश, आयरिश, जिप्सी या आयरिश ट्रैवलर, रोमा या किसी अन्य श्वेत पृष्ठभूमि) के रूप में पहचानता है.',
'Census 2021 से. स्थानीय पड़ोस (LSOA) की आबादी का प्रतिशत जो खुद को श्वेत (अंग्रेज़, वेल्श, स्कॉटिश, उत्तरी आयरिश, ब्रिटिश, आयरिश, जिप्सी या आयरिश ट्रैवलर, रोमा या किसी अन्य श्वेत पृष्ठभूमि) के रूप में पहचानता है.',
'% South Asian':
'Census 2021 से. स्थानीय प्राधिकरण की आबादी का प्रतिशत जो खुद को भारतीय, पाकिस्तानी, बांग्लादेशी या किसी अन्य एशियाई पृष्ठभूमि के रूप में पहचानता है.',
'Census 2021 से. स्थानीय पड़ोस (LSOA) की आबादी का प्रतिशत जो खुद को भारतीय, पाकिस्तानी या बांग्लादेशी के रूप में पहचानता है.',
'% Black':
'Census 2021 से. स्थानीय प्राधिकरण की आबादी का प्रतिशत जो खुद को अश्वेत, अश्वेत ब्रिटिश, कैरिबियाई या अफ्रीकी के रूप में पहचानता है.',
'Census 2021 से. स्थानीय पड़ोस (LSOA) की आबादी का प्रतिशत जो खुद को अश्वेत, अश्वेत ब्रिटिश, कैरिबियाई या अफ्रीकी के रूप में पहचानता है.',
'% East Asian':
'Census 2021 से. स्थानीय प्राधिकरण की आबादी का प्रतिशत जो खुद को चीनी के रूप में पहचानता है.',
'Census 2021 से. स्थानीय पड़ोस (LSOA) की आबादी का प्रतिशत जो खुद को चीनी के रूप में पहचानता है.',
'% SE Asian':
'Census 2021 से. स्थानीय प्राधिकरण की आबादी का प्रतिशत जो खुद को अन्य (गैर-चीनी) पूर्वी/दक्षिण-पूर्वी एशियाई (जैसे फिलिपिनो, वियतनामी या थाई) के रूप में पहचानता है.',
'Census 2021 से. स्थानीय पड़ोस (LSOA) की आबादी का प्रतिशत जो खुद को अन्य (गैर-चीनी) पूर्वी/दक्षिण-पूर्वी एशियाई (जैसे फिलिपिनो, वियतनामी या थाई) के रूप में पहचानता है.',
'% Mixed':
'Census 2021 से. स्थानीय प्राधिकरण की आबादी का प्रतिशत जो खुद को मिश्रित या कई जातीय समूहों (श्वेत और अश्वेत कैरिबियाई, श्वेत और अश्वेत अफ्रीकी, श्वेत और एशियाई या अन्य मिश्रित/बहुजातीय पृष्ठभूमि) के रूप में पहचानता है.',
'Census 2021 से. स्थानीय पड़ोस (LSOA) की आबादी का प्रतिशत जो खुद को मिश्रित या कई जातीय समूहों (श्वेत और अश्वेत कैरिबियाई, श्वेत और अश्वेत अफ्रीकी, श्वेत और एशियाई या अन्य मिश्रित/बहुजातीय पृष्ठभूमि) के रूप में पहचानता है.',
'% Other':
'Census 2021 से. स्थानीय प्राधिकरण की आबादी का प्रतिशत जो खुद को किसी अन्य जातीय समूह (अरब या मुख्य श्रेणियों से बाहर किसी अन्य जातीय समूह) के रूप में पहचानता है.',
'Census 2021 से. स्थानीय पड़ोस (LSOA) की आबादी का प्रतिशत जो खुद को किसी अन्य जातीय समूह (अरब या मुख्य श्रेणियों से बाहर किसी अन्य जातीय समूह) के रूप में पहचानता है.',
'Voter turnout (%)':
'जुलाई 2024 के ब्रिटेन के आम चुनाव में वैध मत देने वाले पंजीकृत मतदाताओं का अनुपात. इसकी गणना वैध मतों को मतदाता सूची के आकार से भाग देकर की जाती है. अधिक मतदान आमतौर पर संपन्न क्षेत्रों और कड़ी चुनावी प्रतिस्पर्धा से जुड़ा होता है.',
'% Labour':
@ -648,19 +649,19 @@ export const details: Record<string, Record<string, string>> = {
'Median age':
'A 2021-es Census alapján (TS007A). Az LSOA szokásos lakóinak medián életkora, ötéves korcsoport-számlálásokból lineáris interpolációval számítva. A fiatalabb népességű területek jellemzően városiak, egyetemi városok vagy több családot vonzanak; az idősebb medián értékek jellemzően vidéki és tengerparti területekre jellemzők.',
'% White':
'A 2021-es Census alapján. A helyi hatóság területén fehérként (angol, walesi, skót, észak-ír, brit, ír, cigány vagy ír vándor, roma, vagy bármely más fehér háttér) azonosított népesség százaléka.',
'A 2021-es Census alapján. A helyi környéken (LSOA) fehérként (angol, walesi, skót, észak-ír, brit, ír, cigány vagy ír vándor, roma, vagy bármely más fehér háttér) azonosított népesség százaléka.',
'% South Asian':
'A 2021-es Census alapján. A helyi hatóság területén indiai, pakisztáni, bangladesi vagy bármely más ázsiai háttérként azonosított népesség százaléka.',
'A 2021-es Census alapján. A helyi környéken (LSOA) indiai, pakisztáni vagy bangladesi háttérként azonosított népesség százaléka.',
'% Black':
'A 2021-es Census alapján. A helyi hatóság területén fekete, brit fekete, karibi vagy afrikai háttérként azonosított népesség százaléka.',
'A 2021-es Census alapján. A helyi környéken (LSOA) fekete, brit fekete, karibi vagy afrikai háttérként azonosított népesség százaléka.',
'% East Asian':
'A 2021-es Census alapján. A helyi hatóság területén kínaiként azonosított népesség százaléka.',
'A 2021-es Census alapján. A helyi környéken (LSOA) kínaiként azonosított népesség százaléka.',
'% SE Asian':
'A 2021-es Census alapján. A helyi hatóság területén más (nem kínai) kelet-/délkelet-ázsiai származásúként (pl. filippínó, vietnámi vagy thai) azonosított népesség százaléka.',
'A 2021-es Census alapján. A helyi környéken (LSOA) más (nem kínai) kelet-/délkelet-ázsiai származásúként (pl. filippínó, vietnámi vagy thai) azonosított népesség százaléka.',
'% Mixed':
'A 2021-es Census alapján. A helyi hatóság területén vegyes vagy többes etnikai csoportként (fehér és fekete karibi, fehér és fekete afrikai, fehér és ázsiai, vagy bármely más vegyes vagy többes háttér) azonosított népesség százaléka.',
'A 2021-es Census alapján. A helyi környéken (LSOA) vegyes vagy többes etnikai csoportként (fehér és fekete karibi, fehér és fekete afrikai, fehér és ázsiai, vagy bármely más vegyes vagy többes háttér) azonosított népesség százaléka.',
'% Other':
'A 2021-es Census alapján. A helyi hatóság területén egyéb etnikai csoportként (arab vagy bármely más, a főkategóriák által nem lefedett etnikai csoport) azonosított népesség százaléka.',
'A 2021-es Census alapján. A helyi környéken (LSOA) egyéb etnikai csoportként (arab vagy bármely más, a főkategóriák által nem lefedett etnikai csoport) azonosított népesség százaléka.',
'Voter turnout (%)':
'A regisztrált szavazók azon aránya, akik érvényes szavazatot adtak le a 2024. júliusi brit parlamenti választáson. Az érvényes szavazatok száma osztva a választói névjegyzékben szereplők számával. A magasabb részvétel általában a tehetősebb területekkel és a szorosabb versenyekkel korrelál.',
'% Labour':

View file

@ -746,6 +746,10 @@ const de: Translations = {
noChangeTitle: 'Nur direkte Verbindungen',
noChangeDesc:
'Beschränkt auf Fahrten <strong>ohne Umstiege</strong>: hinlaufen, ein Verkehrsmittel nehmen, am Ziel weiterlaufen.',
oneChange: 'Ein Umstieg',
oneChangeTitle: 'Fahrten mit einem Umstieg',
oneChangeDesc:
'Erlaubt Fahrten mit <strong>höchstens einem Umstieg</strong>: ein einzelner Wechsel zwischen zwei Verkehrsmitteln ist erlaubt, mehr nicht. Ein Mittelweg zwischen direkten und uneingeschränkten Fahrten.',
noBuses: 'Ohne Bus',
noBusesTitle: 'Busse ausschließen',
noBusesDesc:
@ -1546,22 +1550,22 @@ const de: Translations = {
'Air Quality and Road Safety Score': 'Wert für Luftqualität und Verkehrssicherheit',
// ─ Feature names (Crime) ─
'Serious crime (avg/yr)': 'Schwere Straftaten (Ø/Jahr)',
'Minor crime (avg/yr)': 'Leichte Straftaten (Ø/Jahr)',
'Violence and sexual offences (avg/yr)': 'Gewalt- und Sexualdelikte (Ø/Jahr)',
'Burglary (avg/yr)': 'Einbrüche (Ø/Jahr)',
'Robbery (avg/yr)': 'Raubüberfälle (Ø/Jahr)',
'Vehicle crime (avg/yr)': 'Fahrzeugkriminalität (Ø/Jahr)',
'Anti-social behaviour (avg/yr)': 'Antisoziales Verhalten (Ø/Jahr)',
'Criminal damage and arson (avg/yr)': 'Sachbeschädigung und Brandstiftung (Ø/Jahr)',
'Other theft (avg/yr)': 'Sonstiger Diebstahl (Ø/Jahr)',
'Theft from the person (avg/yr)': 'Diebstahl von der Person (Ø/Jahr)',
'Shoplifting (avg/yr)': 'Ladendiebstahl (Ø/Jahr)',
'Bicycle theft (avg/yr)': 'Fahrraddiebstahl (Ø/Jahr)',
'Drugs (avg/yr)': 'Drogendelikte (Ø/Jahr)',
'Possession of weapons (avg/yr)': 'Waffenbesitz (Ø/Jahr)',
'Public order (avg/yr)': 'Störung der öffentlichen Ordnung (Ø/Jahr)',
'Other crime (avg/yr)': 'Sonstige Straftaten (Ø/Jahr)',
'Serious crime (avg/yr)': 'Schwere Straftaten (Dichte)',
'Minor crime (avg/yr)': 'Leichte Straftaten (Dichte)',
'Violence and sexual offences (avg/yr)': 'Gewalt- und Sexualdelikte (Dichte)',
'Burglary (avg/yr)': 'Einbrüche (Dichte)',
'Robbery (avg/yr)': 'Raubüberfälle (Dichte)',
'Vehicle crime (avg/yr)': 'Fahrzeugkriminalität (Dichte)',
'Anti-social behaviour (avg/yr)': 'Antisoziales Verhalten (Dichte)',
'Criminal damage and arson (avg/yr)': 'Sachbeschädigung und Brandstiftung (Dichte)',
'Other theft (avg/yr)': 'Sonstiger Diebstahl (Dichte)',
'Theft from the person (avg/yr)': 'Diebstahl von der Person (Dichte)',
'Shoplifting (avg/yr)': 'Ladendiebstahl (Dichte)',
'Bicycle theft (avg/yr)': 'Fahrraddiebstahl (Dichte)',
'Drugs (avg/yr)': 'Drogendelikte (Dichte)',
'Possession of weapons (avg/yr)': 'Waffenbesitz (Dichte)',
'Public order (avg/yr)': 'Störung der öffentlichen Ordnung (Dichte)',
'Other crime (avg/yr)': 'Sonstige Straftaten (Dichte)',
// ─ Feature names (Neighbours) ─
'Median age': 'Medianalter',

View file

@ -732,6 +732,10 @@ const en = {
noChangeTitle: 'No-change journeys only',
noChangeDesc:
'Restricts to journeys with <strong>no transfers</strong> —walk, board one transit service, then walk to the destination. Useful when you want a single straight-through commute.',
oneChange: 'One change',
oneChangeTitle: 'One-change journeys',
oneChangeDesc:
'Allows journeys with <strong>at most one transfer</strong> —so a single change between two transit services is fine, but no more. A middle ground between no-change and unrestricted journeys.',
noBuses: 'No buses',
noBusesTitle: 'Excluding buses',
noBusesDesc:
@ -1518,22 +1522,22 @@ const en = {
'Air Quality and Road Safety Score': 'Air Quality and Road Safety Score',
// ─ Feature names (Crime) ─
'Serious crime (avg/yr)': 'Serious crime (avg/yr)',
'Minor crime (avg/yr)': 'Minor crime (avg/yr)',
'Violence and sexual offences (avg/yr)': 'Violence and sexual offences (avg/yr)',
'Burglary (avg/yr)': 'Burglary (avg/yr)',
'Robbery (avg/yr)': 'Robbery (avg/yr)',
'Vehicle crime (avg/yr)': 'Vehicle crime (avg/yr)',
'Anti-social behaviour (avg/yr)': 'Anti-social behaviour (avg/yr)',
'Criminal damage and arson (avg/yr)': 'Criminal damage and arson (avg/yr)',
'Other theft (avg/yr)': 'Other theft (avg/yr)',
'Theft from the person (avg/yr)': 'Theft from the person (avg/yr)',
'Shoplifting (avg/yr)': 'Shoplifting (avg/yr)',
'Bicycle theft (avg/yr)': 'Bicycle theft (avg/yr)',
'Drugs (avg/yr)': 'Drugs (avg/yr)',
'Possession of weapons (avg/yr)': 'Possession of weapons (avg/yr)',
'Public order (avg/yr)': 'Public order (avg/yr)',
'Other crime (avg/yr)': 'Other crime (avg/yr)',
'Serious crime (avg/yr)': 'Serious crime (density)',
'Minor crime (avg/yr)': 'Minor crime (density)',
'Violence and sexual offences (avg/yr)': 'Violence and sexual offences (density)',
'Burglary (avg/yr)': 'Burglary (density)',
'Robbery (avg/yr)': 'Robbery (density)',
'Vehicle crime (avg/yr)': 'Vehicle crime (density)',
'Anti-social behaviour (avg/yr)': 'Anti-social behaviour (density)',
'Criminal damage and arson (avg/yr)': 'Criminal damage and arson (density)',
'Other theft (avg/yr)': 'Other theft (density)',
'Theft from the person (avg/yr)': 'Theft from the person (density)',
'Shoplifting (avg/yr)': 'Shoplifting (density)',
'Bicycle theft (avg/yr)': 'Bicycle theft (density)',
'Drugs (avg/yr)': 'Drugs (density)',
'Possession of weapons (avg/yr)': 'Possession of weapons (density)',
'Public order (avg/yr)': 'Public order (density)',
'Other crime (avg/yr)': 'Other crime (density)',
// ─ Feature names (Neighbours) ─
'Median age': 'Median age',

View file

@ -760,6 +760,10 @@ const fr: Translations = {
noChangeTitle: 'Trajets directs uniquement',
noChangeDesc:
'Limite aux trajets <strong>sans correspondance</strong> : marche, un seul service de transport en commun, puis marche jusquà destination.',
oneChange: 'Une correspondance',
oneChangeTitle: 'Trajets avec une correspondance',
oneChangeDesc:
'Autorise les trajets avec <strong>au plus une correspondance</strong> : un seul changement entre deux services est permis, pas davantage. Un compromis entre les trajets directs et sans restriction.',
noBuses: 'Sans bus',
noBusesTitle: 'Exclure les bus',
noBusesDesc:
@ -893,7 +897,7 @@ const fr: Translations = {
walk: 'Marche',
cycle: 'Vélo',
nationalAvg: 'Moyenne nationale',
crimeDataEnds: 'Les données de police pour cette zone s\'arrêtent en {{year}}',
crimeDataEnds: "Les données de police pour cette zone s'arrêtent en {{year}}",
},
// ── Street View ────────────────────────────────────
@ -1563,22 +1567,22 @@ const fr: Translations = {
'Air Quality and Road Safety Score': 'Score qualité de lair et sécurité routière',
// ─ Feature names (Crime) ─
'Serious crime (avg/yr)': 'Infractions graves (moy./an)',
'Minor crime (avg/yr)': 'Infractions mineures (moy./an)',
'Violence and sexual offences (avg/yr)': 'Violences et infractions sexuelles (moy./an)',
'Burglary (avg/yr)': 'Cambriolages (moy./an)',
'Robbery (avg/yr)': 'Vols avec violence (moy./an)',
'Vehicle crime (avg/yr)': 'Infractions liées aux véhicules (moy./an)',
'Anti-social behaviour (avg/yr)': 'Comportements antisociaux (moy./an)',
'Criminal damage and arson (avg/yr)': 'Dégradations et incendies criminels (moy./an)',
'Other theft (avg/yr)': 'Autres vols (moy./an)',
'Theft from the person (avg/yr)': 'Vols à la personne (moy./an)',
'Shoplifting (avg/yr)': 'Vols à létalage (moy./an)',
'Bicycle theft (avg/yr)': 'Vols de vélos (moy./an)',
'Drugs (avg/yr)': 'Infractions liées aux stupéfiants (moy./an)',
'Possession of weapons (avg/yr)': 'Possession darmes (moy./an)',
'Public order (avg/yr)': 'Troubles à lordre public (moy./an)',
'Other crime (avg/yr)': 'Autres infractions (moy./an)',
'Serious crime (avg/yr)': 'Infractions graves (densité)',
'Minor crime (avg/yr)': 'Infractions mineures (densité)',
'Violence and sexual offences (avg/yr)': 'Violences et infractions sexuelles (densité)',
'Burglary (avg/yr)': 'Cambriolages (densité)',
'Robbery (avg/yr)': 'Vols avec violence (densité)',
'Vehicle crime (avg/yr)': 'Infractions liées aux véhicules (densité)',
'Anti-social behaviour (avg/yr)': 'Comportements antisociaux (densité)',
'Criminal damage and arson (avg/yr)': 'Dégradations et incendies criminels (densité)',
'Other theft (avg/yr)': 'Autres vols (densité)',
'Theft from the person (avg/yr)': 'Vols à la personne (densité)',
'Shoplifting (avg/yr)': 'Vols à létalage (densité)',
'Bicycle theft (avg/yr)': 'Vols de vélos (densité)',
'Drugs (avg/yr)': 'Infractions liées aux stupéfiants (densité)',
'Possession of weapons (avg/yr)': 'Possession darmes (densité)',
'Public order (avg/yr)': 'Troubles à lordre public (densité)',
'Other crime (avg/yr)': 'Autres infractions (densité)',
// ─ Feature names (Neighbours) ─
'Median age': 'Âge médian',

View file

@ -725,6 +725,10 @@ const hi: Translations = {
noChangeTitle: 'केवल सीधी यात्राएँ',
noChangeDesc:
'<strong>बिना बदलाव</strong> वाली यात्राओं तक सीमित — पैदल चलें, एक सार्वजनिक परिवहन सेवा लें, फिर गंतव्य तक पैदल जाएं. सीधे आवागमन के लिए उपयोगी.',
oneChange: 'एक बदलाव',
oneChangeTitle: 'एक बदलाव वाली यात्राएँ',
oneChangeDesc:
'<strong>अधिकतम एक बदलाव</strong> वाली यात्राओं की अनुमति देता है — दो परिवहन सेवाओं के बीच एक बदलाव ठीक है, इससे अधिक नहीं. बिना बदलाव और असीमित यात्राओं के बीच का विकल्प.',
noBuses: 'बसों के बिना',
noBusesTitle: 'बसों को बाहर रखें',
noBusesDesc:
@ -1468,22 +1472,22 @@ const hi: Translations = {
'Health Deprivation and Disability Score': 'स्वास्थ्य वंचना और विकलांगता स्कोर',
'Housing Conditions Score': 'आवास स्थिति स्कोर',
'Air Quality and Road Safety Score': 'हवा की गुणवत्ता और सड़क सुरक्षा स्कोर',
'Serious crime (avg/yr)': 'गंभीर अपराध (औसत/वर्ष)',
'Minor crime (avg/yr)': 'मामूली अपराध (औसत/वर्ष)',
'Violence and sexual offences (avg/yr)': 'हिंसा और यौन अपराध (औसत/वर्ष)',
'Burglary (avg/yr)': 'सेंधमारी (औसत/वर्ष)',
'Robbery (avg/yr)': 'लूट (औसत/वर्ष)',
'Vehicle crime (avg/yr)': 'वाहन अपराध (औसत/वर्ष)',
'Anti-social behaviour (avg/yr)': 'असामाजिक व्यवहार (औसत/वर्ष)',
'Criminal damage and arson (avg/yr)': 'आपराधिक क्षति और आगजनी (औसत/वर्ष)',
'Other theft (avg/yr)': 'अन्य चोरी (औसत/वर्ष)',
'Theft from the person (avg/yr)': 'व्यक्ति से चोरी (औसत/वर्ष)',
'Shoplifting (avg/yr)': 'दुकान से चोरी (औसत/वर्ष)',
'Bicycle theft (avg/yr)': 'साइकिल चोरी (औसत/वर्ष)',
'Drugs (avg/yr)': 'ड्रग्स (औसत/वर्ष)',
'Possession of weapons (avg/yr)': 'हथियार रखने के अपराध (औसत/वर्ष)',
'Public order (avg/yr)': 'सार्वजनिक व्यवस्था अपराध (औसत/वर्ष)',
'Other crime (avg/yr)': 'अन्य अपराध (औसत/वर्ष)',
'Serious crime (avg/yr)': 'गंभीर अपराध (घनत्व)',
'Minor crime (avg/yr)': 'मामूली अपराध (घनत्व)',
'Violence and sexual offences (avg/yr)': 'हिंसा और यौन अपराध (घनत्व)',
'Burglary (avg/yr)': 'सेंधमारी (घनत्व)',
'Robbery (avg/yr)': 'लूट (घनत्व)',
'Vehicle crime (avg/yr)': 'वाहन अपराध (घनत्व)',
'Anti-social behaviour (avg/yr)': 'असामाजिक व्यवहार (घनत्व)',
'Criminal damage and arson (avg/yr)': 'आपराधिक क्षति और आगजनी (घनत्व)',
'Other theft (avg/yr)': 'अन्य चोरी (घनत्व)',
'Theft from the person (avg/yr)': 'व्यक्ति से चोरी (घनत्व)',
'Shoplifting (avg/yr)': 'दुकान से चोरी (घनत्व)',
'Bicycle theft (avg/yr)': 'साइकिल चोरी (घनत्व)',
'Drugs (avg/yr)': 'ड्रग्स (घनत्व)',
'Possession of weapons (avg/yr)': 'हथियार रखने के अपराध (घनत्व)',
'Public order (avg/yr)': 'सार्वजनिक व्यवस्था अपराध (घनत्व)',
'Other crime (avg/yr)': 'अन्य अपराध (घनत्व)',
'Median age': 'मध्य आयु',
'% White': '% श्वेत',
'% South Asian': '% दक्षिण एशियाई',

View file

@ -749,6 +749,10 @@ const hu: Translations = {
noChangeTitle: 'Csak közvetlen járatok',
noChangeDesc:
'<strong>Átszállás nélküli</strong> utazásokra korlátoz: gyaloglás, egy közlekedési eszköz, majd gyaloglás a célig.',
oneChange: 'Egy átszállás',
oneChangeTitle: 'Egy átszállásos utazások',
oneChangeDesc:
'<strong>Legfeljebb egy átszállást</strong> engedélyez: két közlekedési eszköz közötti egyetlen váltás megengedett, több nem. Köztes megoldás az átszállás nélküli és a korlátozás nélküli utazások között.',
noBuses: 'Busz nélkül',
noBusesTitle: 'Buszok kizárása',
noBusesDesc:
@ -1542,22 +1546,22 @@ const hu: Translations = {
'Air Quality and Road Safety Score': 'Levegőminőség és közlekedésbiztonság pontszám',
// ─ Feature names (Crime) ─
'Serious crime (avg/yr)': 'Súlyos bűncselekmény (éves átlag)',
'Minor crime (avg/yr)': 'Kisebb bűncselekmény (éves átlag)',
'Violence and sexual offences (avg/yr)': 'Erőszak és szexuális bűncselekmények (éves átlag)',
'Burglary (avg/yr)': 'Betörés (éves átlag)',
'Robbery (avg/yr)': 'Rablás (éves átlag)',
'Vehicle crime (avg/yr)': 'Járművel kapcsolatos bűncselekmények (éves átlag)',
'Anti-social behaviour (avg/yr)': 'Közösségellenes magatartás (éves átlag)',
'Criminal damage and arson (avg/yr)': 'Rongálás és gyújtogatás (éves átlag)',
'Other theft (avg/yr)': 'Egyéb lopás (éves átlag)',
'Theft from the person (avg/yr)': 'Személytől történő lopás (éves átlag)',
'Shoplifting (avg/yr)': 'Bolti lopás (éves átlag)',
'Bicycle theft (avg/yr)': 'Kerékpárlopás (éves átlag)',
'Drugs (avg/yr)': 'Kábítószer (éves átlag)',
'Possession of weapons (avg/yr)': 'Fegyverbirtoklás (éves átlag)',
'Public order (avg/yr)': 'Közrend (éves átlag)',
'Other crime (avg/yr)': 'Egyéb bűncselekmény (éves átlag)',
'Serious crime (avg/yr)': 'Súlyos bűncselekmény (sűrűség)',
'Minor crime (avg/yr)': 'Kisebb bűncselekmény (sűrűség)',
'Violence and sexual offences (avg/yr)': 'Erőszak és szexuális bűncselekmények (sűrűség)',
'Burglary (avg/yr)': 'Betörés (sűrűség)',
'Robbery (avg/yr)': 'Rablás (sűrűség)',
'Vehicle crime (avg/yr)': 'Járművel kapcsolatos bűncselekmények (sűrűség)',
'Anti-social behaviour (avg/yr)': 'Közösségellenes magatartás (sűrűség)',
'Criminal damage and arson (avg/yr)': 'Rongálás és gyújtogatás (sűrűség)',
'Other theft (avg/yr)': 'Egyéb lopás (sűrűség)',
'Theft from the person (avg/yr)': 'Személytől történő lopás (sűrűség)',
'Shoplifting (avg/yr)': 'Bolti lopás (sűrűség)',
'Bicycle theft (avg/yr)': 'Kerékpárlopás (sűrűség)',
'Drugs (avg/yr)': 'Kábítószer (sűrűség)',
'Possession of weapons (avg/yr)': 'Fegyverbirtoklás (sűrűség)',
'Public order (avg/yr)': 'Közrend (sűrűség)',
'Other crime (avg/yr)': 'Egyéb bűncselekmény (sűrűség)',
// ─ Feature names (Neighbours) ─
'Median age': 'Medián életkor',

View file

@ -695,6 +695,10 @@ const zh: Translations = {
noChangeTitle: '仅不换乘行程',
noChangeDesc:
'仅限<strong>不换乘</strong>行程:步行、乘坐一次公共交通,再步行到目的地。适合希望一路直达的通勤。',
oneChange: '一次换乘',
oneChangeTitle: '一次换乘行程',
oneChangeDesc:
'允许<strong>最多换乘一次</strong>的行程:在两种公共交通之间换乘一次可以,但不能更多。介于不换乘与不限换乘之间的折中选择。',
noBuses: '不坐公交',
noBusesTitle: '排除公交',
noBusesDesc:
@ -1453,22 +1457,22 @@ const zh: Translations = {
'Air Quality and Road Safety Score': '空气质量与道路安全得分',
// ─ Feature names (Crime) ─
'Serious crime (avg/yr)': '严重犯罪(年均',
'Minor crime (avg/yr)': '轻微犯罪(年均',
'Violence and sexual offences (avg/yr)': '暴力和性犯罪(年均',
'Burglary (avg/yr)': '入室盗窃(年均',
'Robbery (avg/yr)': '抢劫(年均',
'Vehicle crime (avg/yr)': '车辆犯罪(年均',
'Anti-social behaviour (avg/yr)': '反社会行为(年均',
'Criminal damage and arson (avg/yr)': '刑事毁坏和纵火(年均',
'Other theft (avg/yr)': '其他盗窃(年均',
'Theft from the person (avg/yr)': '人身盗窃(年均',
'Shoplifting (avg/yr)': '商店盗窃(年均',
'Bicycle theft (avg/yr)': '自行车盗窃(年均',
'Drugs (avg/yr)': '毒品犯罪(年均',
'Possession of weapons (avg/yr)': '非法持有武器(年均',
'Public order (avg/yr)': '扰乱公共秩序(年均',
'Other crime (avg/yr)': '其他犯罪(年均',
'Serious crime (avg/yr)': '严重犯罪(密度',
'Minor crime (avg/yr)': '轻微犯罪(密度',
'Violence and sexual offences (avg/yr)': '暴力和性犯罪(密度',
'Burglary (avg/yr)': '入室盗窃(密度',
'Robbery (avg/yr)': '抢劫(密度',
'Vehicle crime (avg/yr)': '车辆犯罪(密度',
'Anti-social behaviour (avg/yr)': '反社会行为(密度',
'Criminal damage and arson (avg/yr)': '刑事毁坏和纵火(密度',
'Other theft (avg/yr)': '其他盗窃(密度',
'Theft from the person (avg/yr)': '人身盗窃(密度',
'Shoplifting (avg/yr)': '商店盗窃(密度',
'Bicycle theft (avg/yr)': '自行车盗窃(密度',
'Drugs (avg/yr)': '毒品犯罪(密度',
'Possession of weapons (avg/yr)': '非法持有武器(密度',
'Public order (avg/yr)': '扰乱公共秩序(密度',
'Other crime (avg/yr)': '其他犯罪(密度',
// ─ Feature names (Neighbours) ─
'Median age': '中位年龄',

View file

@ -237,7 +237,7 @@ export const STACKED_GROUPS: Record<
{
label: 'Serious crime',
feature: 'Serious crime (avg/yr)',
unit: 'avg/yr',
unit: '',
components: [
'Violence and sexual offences (avg/yr)',
'Robbery (avg/yr)',
@ -248,7 +248,7 @@ export const STACKED_GROUPS: Record<
{
label: 'Minor crime',
feature: 'Minor crime (avg/yr)',
unit: 'avg/yr',
unit: '',
components: [
'Anti-social behaviour (avg/yr)',
'Criminal damage and arson (avg/yr)',

View file

@ -100,9 +100,10 @@ export function getSpecificCrimeFilterMeta(features: FeatureMeta[]): FeatureMeta
step: 1,
description:
'Violence, burglary, robbery, drugs, shoplifting, vehicle crime, anti-social behaviour, public order, theft, and other crime types',
detail: 'Filter by one street-level crime category at a time using yearly averages per LSOA.',
detail:
'Filter by one street-level crime category at a time using an area-normalised crime density near each postcode (not a count of incidents per year).',
source: 'crime',
suffix: '/yr',
suffix: '',
};
}

View file

@ -42,10 +42,12 @@ export function dedupeTravelTimeEntries(entries: TravelTimeEntry[]): TravelTimeE
label: existing.label || entry.label,
timeRange: mergeTimeRanges(existing.timeRange, entry.timeRange),
useBest: existing.useBest || entry.useBest,
// noChange/noBuses are part of the dedupe key, so for two entries to
// collide here they must already have matching flags. Carry them through
// explicitly so unset (undefined) doesn't clobber set (false/true).
// noChange/oneChange/noBuses are part of the dedupe key, so for two
// entries to collide here they must already have matching flags. Carry
// them through explicitly so unset (undefined) doesn't clobber set
// (false/true).
noChange: existing.noChange ?? entry.noChange,
oneChange: existing.oneChange ?? entry.oneChange,
noBuses: existing.noBuses ?? entry.noBuses,
};
}

View file

@ -49,6 +49,7 @@ describe('url-state', () => {
timeRange: [0, 30],
useBest: true,
noChange: false,
oneChange: false,
noBuses: false,
},
]);
@ -118,6 +119,7 @@ describe('url-state', () => {
timeRange: [10, 45],
useBest: false,
noChange: false,
oneChange: false,
noBuses: false,
},
]);

View file

@ -304,9 +304,9 @@ export function parseUrlState(): UrlState {
// Travel time: repeated `tt` params
// Format: serverMode:slug:label[:b][:min:max]
// serverMode is one of: car | bicycle | walking | transit | transit-no-bus
// | transit-no-change | transit-no-change-no-bus. transit-one-change[-no-bus]
// variants are server-side only and will cause the entry to be dropped here
// (parseServerMode returns null) so we don't silently broaden the user's filter.
// | transit-no-change | transit-no-change-no-bus | transit-one-change
// | transit-one-change-no-bus. Unknown modes cause the entry to be dropped
// here (parseServerMode returns null) so we don't silently broaden the filter.
const ttParams = params.getAll('tt');
if (ttParams.length > 0) {
const entries: TravelTimeEntry[] = [];
@ -337,6 +337,7 @@ export function parseUrlState(): UrlState {
timeRange,
useBest,
noChange: parsedMode.noChange,
oneChange: parsedMode.oneChange,
noBuses: parsedMode.noBuses,
});
}

View file

@ -52,6 +52,13 @@ JUMP_MIN_PRICE = 2_000_000
MIN_BUILD_YEAR = 1700
MAX_BUILD_YEAR = 2030
# Open-ended "before YYYY" bands (EPC only emits "before 1900") have no lower
# bound. Rather than dropping these mostly Victorian-and-older dwellings, we
# publish a representative year this many years under the stated upper bound
# ("before 1900" -> 1890) so they still carry a build year and survive an
# age-range filter; is_construction_date_approximate flags it as an estimate.
OPEN_BAND_YEARS_BEFORE_BOUND = 10
# Plausibility bounds for raw EPC dimensions. EPC lodgements contain data-entry
# errors (0 m storey heights, 116 m "interior height", 9,210 m² floor areas, 99
# habitable rooms) that otherwise propagate verbatim into the published per-
@ -68,10 +75,13 @@ def epc_band_to_year(band: pl.Expr) -> pl.Expr:
EPC age bands are ranges (e.g. ``1950-1966``); we use the band MIDPOINT
(1958) rather than the lower bound, which previously biased every band-derived
year ~10-15 years too young. Open-ended lower bands (``before 1900``) are too
wide to pin to a year and return null. Single-year / ``... onwards`` bands use
that year. Already-numeric inputs (a year produced by an earlier call) pass
through unchanged. Years outside [MIN_BUILD_YEAR, MAX_BUILD_YEAR] are nulled.
year ~10-15 years too young. Open-ended lower bands (``before 1900``) have no
lower bound, so we publish a representative year OPEN_BAND_YEARS_BEFORE_BOUND
years under the stated upper bound (``before 1900`` -> 1890) rather than
dropping the dwelling; is_construction_date_approximate flags it as an
estimate. Single-year / ``... onwards`` bands use that year. Already-numeric
inputs (a year produced by an earlier call) pass through unchanged. Years
outside [MIN_BUILD_YEAR, MAX_BUILD_YEAR] are nulled.
"""
text = (
band.cast(pl.Utf8)
@ -82,7 +92,7 @@ def epc_band_to_year(band: pl.Expr) -> pl.Expr:
high = text.str.extract(r"(\d{4})\D+(\d{4})", 2).cast(pl.Int32, strict=False)
year = (
pl.when(text.str.starts_with("before "))
.then(None)
.then(low - OPEN_BAND_YEARS_BEFORE_BOUND)
.when(high.is_not_null())
.then(((low + high) / 2).round(0).cast(pl.Int32))
.otherwise(low)

View file

@ -1149,8 +1149,9 @@ def _canonical_epc_property_type_expr() -> pl.Expr:
def _construction_year_expr(column: str = "construction_age_band") -> pl.Expr:
# Use the shared band->midpoint-year mapping so the direct-EPC / listings
# path matches join_epc_pp (band midpoint, not lower bound; 'before 1900' and
# implausible years -> null). Already-numeric inputs pass through unchanged.
# path matches join_epc_pp (band midpoint, not lower bound; 'before 1900' ->
# 1890 representative year; implausible years -> null). Already-numeric inputs
# pass through unchanged.
return epc_band_to_year(pl.col(column))
@ -1728,7 +1729,7 @@ def _property_match_candidate_frame(wide: pl.LazyFrame) -> pl.DataFrame:
def _index_candidates(
candidates: pl.DataFrame, postcode_key: str, uprn_key: str
candidates: pl.DataFrame, postcode_key: str, uprn_key: str, address_key: str
) -> tuple[dict[str, list[dict]], dict[str, dict]]:
"""Index candidate rows for matching, in a single pass over the frame.
@ -1736,17 +1737,41 @@ def _index_candidates(
fuzzy street-address match; the UPRN index drives the exact match and is
postcode-independent, so it still resolves when a listing's postcode is
slightly off.
The EPC register's UPRN is NOT unique: a single building/parent UPRN fans
across many distinct flats (up to 58 distinct (address, postcode) rows in
the 2026-06 data; ~9k UPRNs collide, touching ~20k epc_pp rows). Such a
UPRN cannot serve as a 1:1 exact-match key it would mis-link a listing to
one arbitrary flat so any UPRN that resolves to more than one distinct
``(postcode_key, address_key)`` identity is dropped from ``uprn_index``;
those listings fall back to the fuzzy street-address matcher, which
disambiguates the specific flat. A UPRN repeated for the SAME identity
(one genuine property) is kept.
"""
buckets: dict[str, list[dict]] = {}
uprn_index: dict[str, dict] = {}
# uprn -> first_row, plus the identity of that first row; a uprn drops out
# the moment a second distinct (postcode, address) identity appears.
uprn_first: dict[str, dict] = {}
uprn_identity: dict[str, tuple] = {}
uprn_dropped: set[str] = set()
for row in candidates.iter_rows(named=True):
postcode = row.get(postcode_key)
if postcode:
buckets.setdefault(postcode, []).append(row)
uprn = _normalize_uprn(row.get(uprn_key))
if uprn and uprn not in uprn_index:
uprn_index[uprn] = row
return buckets, uprn_index
if not uprn or uprn in uprn_dropped:
continue
identity = (row.get(postcode_key), row.get(address_key))
if uprn not in uprn_first:
uprn_first[uprn] = row
uprn_identity[uprn] = identity
elif identity != uprn_identity[uprn]:
# Same UPRN, different (postcode, address): a non-unique parent/
# building UPRN. Remove it so it cannot act as a 1:1 key.
del uprn_first[uprn]
del uprn_identity[uprn]
uprn_dropped.add(uprn)
return buckets, uprn_first
def _best_listing_property_candidate(
@ -1783,7 +1808,10 @@ def _match_listing_properties(
return _empty_listing_property_matches()
buckets, uprn_index = _index_candidates(
property_candidates, "_property_match_postcode", "uprn"
property_candidates,
"_property_match_postcode",
"uprn",
"_property_match_address",
)
best_matches = []
for listing in listing_matches.iter_rows(named=True):
@ -1909,7 +1937,10 @@ def _match_direct_epc(
return _empty_direct_epc_matches()
buckets, uprn_index = _index_candidates(
epc_candidates, "_direct_epc_match_postcode", "_direct_epc_uprn"
epc_candidates,
"_direct_epc_match_postcode",
"_direct_epc_uprn",
"_direct_epc_match_address",
)
street_index, noise_tokens = _index_epc_streets(epc_candidates)
street_score_cache: dict[tuple[str, str], list[tuple[int, str]]] = {}
@ -2214,6 +2245,25 @@ class _BuildResult:
listings: pl.DataFrame | None = None
# Property-level Yes/No flags default to "No" once all EPC + listings overlays
# have been coalesced. A property with no EPC match has no recorded social
# tenure / listed status, which is "No", not "unknown": join_epc_pp fills
# was_council_house with "No" only for EPC-matched rows (it runs before the
# fuzzy join), so without this the ~32% of EPC-unmatched properties would
# publish null instead of "No".
_PROPERTY_LEVEL_NO_DEFAULT_COLUMNS = (LISTED_BUILDING_FEATURE, "was_council_house")
def _fill_property_level_no_defaults(frame: pl.LazyFrame) -> pl.LazyFrame:
"""Default the property-level Yes/No flag columns to "No" where null."""
return frame.with_columns(
*(
pl.col(column).fill_null("No")
for column in _PROPERTY_LEVEL_NO_DEFAULT_COLUMNS
)
)
def _build(
epc_pp_path: Path,
arcgis_path: Path,
@ -2303,7 +2353,12 @@ def _build(
)
wide = _filter_to_active_english_postcodes(wide, active_postcodes)
wide = wide.with_columns(pl.col(LISTED_BUILDING_FEATURE).fill_null("No"))
# Default property-level Yes/No flags to "No" here: after the listings
# overlay coalesce (so a directly-matched "Yes" survives) and before the
# rename to "Former council house". This is the single place the FINAL
# was_council_house column (null for ~32% EPC-unmatched rows) gets its "No"
# default, alongside Listed building.
wide = _fill_property_level_no_defaults(wide)
# NSPL Feb 2026 renamed geographic code columns to {field}{year}cd.
# `_active_english_postcode_area` aliases them back to the short canonical

View file

@ -774,7 +774,7 @@ def test_epc_band_to_year_uses_midpoint_and_clamps():
"b": [
"England and Wales: 1950-1966", # midpoint 1958
"1900-1929", # midpoint 1914
"England and Wales: before 1900", # too wide -> null
"England and Wales: before 1900", # open-ended -> 1900 - 10 = 1890
"2012 onwards", # single year
"1012", # implausible -> null
"2202", # implausible -> null
@ -784,4 +784,4 @@ def test_epc_band_to_year_uses_midpoint_and_clamps():
}
)
years = df.select(epc_band_to_year(pl.col("b")).alias("y"))["y"].to_list()
assert years == [1958, 1914, None, 2012, None, None, None, 1958]
assert years == [1958, 1914, 1890, 2012, None, None, None, 1958]

View file

@ -16,6 +16,7 @@ from pipeline.transform.merge import (
_best_listing_match,
_coalesce_direct_epc_columns,
_dedupe_collapsed_properties,
_fill_property_level_no_defaults,
_filter_to_active_english_postcodes,
_join_area_side_tables,
_finalize_listings,
@ -943,6 +944,61 @@ def test_match_direct_epc_matches_by_uprn_across_postcodes() -> None:
assert matches["_direct_epc_match_method"].to_list() == ["uprn"]
def test_match_direct_epc_ignores_nonunique_building_uprn() -> None:
# A parent/building UPRN that resolves to several distinct (address,
# postcode) flats cannot act as a 1:1 exact-match key — it would mis-link
# the listing to one arbitrary flat. The listing must fall through to the
# street-address matcher, which resolves the specific flat.
matches = _match_direct_epc(
_listing_matches(
[
{
"_listing_uprn": "100000000001",
"_listing_match_address": "FLAT 2 EXAMPLE COURT",
"_listing_match_postcode": "AA11AA",
}
]
),
_direct_epc_candidates(
[
{
"_direct_epc_uprn": "100000000001",
"_direct_epc_match_address": "FLAT 1 EXAMPLE COURT",
"_direct_epc_address": "Flat 1, Example Court",
},
{
"_direct_epc_uprn": "100000000001",
"_direct_epc_match_address": "FLAT 2 EXAMPLE COURT",
"_direct_epc_address": "Flat 2, Example Court",
},
]
),
)
assert matches.height == 1
# Resolved by address, not the ambiguous building UPRN, and to the right flat.
assert matches["_direct_epc_match_method"].to_list() != ["uprn"]
assert matches["_direct_epc_address"].to_list() == ["Flat 2, Example Court"]
def test_fill_property_level_no_defaults_defaults_unmatched_to_no() -> None:
# EPC-unmatched properties arrive with null was_council_house / Listed
# building; they must publish "No" (not null), while a genuine "Yes" is
# preserved.
frame = pl.LazyFrame(
{
LISTED_BUILDING_FEATURE: ["Yes", None, "No"],
"was_council_house": ["Yes", None, "No"],
}
)
out = _fill_property_level_no_defaults(frame).collect()
assert out["was_council_house"].to_list() == ["Yes", "No", "No"]
assert out[LISTED_BUILDING_FEATURE].to_list() == ["Yes", "No", "No"]
assert out["was_council_house"].null_count() == 0
def test_match_direct_epc_matches_by_address_in_same_postcode() -> None:
matches = _match_direct_epc(
_listing_matches([{"_listing_match_address": "1 EXAMPLE ROAD"}]),

View file

@ -3,9 +3,12 @@ import json
import polars as pl
from pipeline.transform.transform_poi import (
BUS_STOP_DEDUP_RADIUS_M,
_load_ofsted_ratings,
_school_icon_category_expr,
_significant_tokens,
osm_groceries_colocated_with_geolytix,
osm_stops_near_naptan,
transform,
transform_grocery_retail_points,
)
@ -71,6 +74,61 @@ def test_osm_groceries_colocated_with_geolytix_handles_empty_inputs():
assert osm_groceries_colocated_with_geolytix(osm, empty_glx) == []
def test_significant_tokens_keeps_ms_short_brand():
# "M&S" strips to "ms" (len 2); without the short-brand allowlist it would
# tokenise to set() and never dedupe against the GEOLYTIX "M&S" chain.
assert _significant_tokens("M&S") == {"ms"}
assert _significant_tokens("M&S Simply Food") == {"ms", "simply", "food"}
# Other sub-3-char tokens stay excluded (allowlist must remain minimal).
assert _significant_tokens("Co") == set()
def test_osm_groceries_colocated_with_geolytix_dedupes_m_and_s():
# GEOLYTIX brand "M&S" must now match a colocated OSM "M&S Simply Food".
geolytix = pl.DataFrame({"category": ["M&S"], "lat": [53.0], "lng": [-1.5]})
osm = pl.DataFrame(
{
"id": ["ms-dup"],
"name": ["M&S Simply Food"],
"lat": [53.00001],
"lng": [-1.5],
}
)
assert osm_groceries_colocated_with_geolytix(osm, geolytix, radius_m=50.0) == [
"ms-dup"
]
def test_osm_groceries_colocated_default_radius_catches_60m_offset():
# OSM and GEOLYTIX routinely place the same store 50-100m apart. The widened
# 100m default catches a ~60m same-brand duplicate that the old 50m missed.
geolytix = pl.DataFrame({"category": ["Tesco"], "lat": [53.0], "lng": [-1.5]})
osm = pl.DataFrame(
# ~60 m north of the GEOLYTIX point.
{"id": ["dup"], "name": ["Tesco Express"], "lat": [53.00054], "lng": [-1.5]}
)
assert osm_groceries_colocated_with_geolytix(osm, geolytix) == ["dup"]
assert osm_groceries_colocated_with_geolytix(osm, geolytix, radius_m=50.0) == []
def test_osm_stops_near_naptan_drops_within_radius_keeps_far():
# A ~60m-offset OSM platform duplicates the NaPTAN stop (dropped at the 100m
# default); a genuine opposite-direction stop ~160m away is kept.
naptan = pl.DataFrame({"id": ["n1"], "lat": [51.5000], "lng": [-0.1000]})
osm = pl.DataFrame(
{
"id": ["dup", "far"],
# ~60 m and ~160 m north of the NaPTAN stop.
"lat": [51.50054, 51.50144],
"lng": [-0.1000, -0.1000],
}
)
assert BUS_STOP_DEDUP_RADIUS_M == 100.0
assert osm_stops_near_naptan(osm, naptan) == ["dup"]
# At the old 50m radius the duplicate survived (the bug this fix closes).
assert osm_stops_near_naptan(osm, naptan, radius_m=50.0) == []
def _write_boundary(tmp_path):
"""A FeatureCollection whose single feature covers the London-area test
coords used by the transform() fixtures, so in_england_mask keeps them."""
@ -505,6 +563,32 @@ def test_osm_supermarkets_dropped(tmp_path):
assert convenience.height == 1
def test_transform_relabels_osm_iceland_to_brand(tmp_path):
# OSM tags Iceland as shop/frozen_food -> "Deli & Specialty". A genuine
# Iceland store GEOLYTIX lacks (away from any GEOLYTIX point) must survive
# AND be relabelled to the "Iceland" brand, not ship mislabelled as a deli.
raw = pl.DataFrame(
{
"id": ["ice1"],
"name": ["Iceland"],
"category": ["shop/frozen_food"],
"lat": [51.40],
"lng": [-0.05],
}
)
inputs = _write_transform_inputs(tmp_path, raw)
out = transform(**inputs).collect()
row = out.filter(pl.col("name") == "Iceland")
assert row.height == 1
assert row["group"][0] == "Groceries"
assert row["category"][0] == "Iceland"
assert row["icon_category"][0] == "Iceland"
# No deli pin remains for the relabelled store.
assert out.filter(pl.col("category") == "Deli & Specialty").height == 0
def test_transform_grocery_dedup_drops_only_grocery_aspect(tmp_path):
# The _write_transform_inputs fixture seeds 5 GEOLYTIX "Tesco" points at
# (51.52, -0.14). An OSM object colocated there carrying "Tesco" in its name

View file

@ -1579,8 +1579,11 @@ def transform_gias_schools(
# Store". GEOLYTIX is authoritative for its chains, so an OSM grocery row that
# sits on top of a GEOLYTIX point AND carries that point's brand name is the
# same physical store and is dropped. Independent corner shops never carry a
# chain brand, so they are kept.
GROCERY_DEDUP_RADIUS_M = 50.0
# chain brand, so they are kept. 100m (not 50m) because OSM and GEOLYTIX
# routinely place the same branded store 50-100m apart (entrance vs centroid vs
# car park); at 50m ~166 same-brand duplicates (Iceland, Morrisons Daily, Tesco
# Express, Sainsbury's Local, ...) survived just outside the radius.
GROCERY_DEDUP_RADIUS_M = 100.0
# Brand-token aliases so an OSM name spelt differently from the GEOLYTIX brand
# still matches. GEOLYTIX's "Co-op" tokenises to "coop", but OSM frequently
@ -1591,15 +1594,34 @@ _GROCERY_TOKEN_ALIASES = {
"cooperatives": "coop",
}
# Brand tokens shorter than the 3-char floor that must still match. "M&S" strips
# to "ms" (len 2) and would otherwise tokenise to set(), so M&S never deduped and
# OSM M&S grocery rows survived as phantom duplicates of the GEOLYTIX M&S chain.
# "M&S" is the only GEOLYTIX brand whose alnum form is <3 chars, so this
# allowlist is safe (no other brand collides).
_GROCERY_SHORT_BRAND_TOKENS = {"ms"}
# OSM tags Iceland/Farmfoods stores shop/frozen_food, which CATEGORY_MAP routes
# to "Deli & Specialty". The grocery dedup drops the ones colocated with a
# GEOLYTIX twin, but genuine stores GEOLYTIX lacks would otherwise survive
# mislabelled as deli, shadowing the brand pill and falling outside the headline
# grocery metric. Relabel the survivors (matched by name) to the brand so they
# line up with the GEOLYTIX rows.
OSM_BRAND_RELABEL = {
"Iceland": "Iceland",
"Farmfoods": "Farmfoods",
}
def _significant_tokens(name: str | None) -> set[str]:
"""Lower-case alphanumeric tokens of length >= 3 from a POI name (aliased)."""
"""Lower-case alphanumeric tokens of length >= 3 from a POI name (aliased),
plus short brand tokens in _GROCERY_SHORT_BRAND_TOKENS (e.g. "ms" for M&S)."""
if not name:
return set()
tokens: set[str] = set()
for raw in str(name).lower().split():
token = "".join(ch for ch in raw if ch.isalnum())
if len(token) >= 3:
if len(token) >= 3 or token in _GROCERY_SHORT_BRAND_TOKENS:
tokens.add(_GROCERY_TOKEN_ALIASES.get(token, token))
return tokens
@ -1608,7 +1630,14 @@ def _significant_tokens(name: str | None) -> set[str]:
# gaps. Where NaPTAN already has a stop within this radius the area is covered,
# so the colocated OSM platform is dropped to avoid double-counting; OSM
# platforms with no nearby NaPTAN stop (the gaps) are kept.
BUS_STOP_DEDUP_RADIUS_M = 50.0
#
# 100 m, not 50 m: at 50 m ~24.6k OSM platforms 50-100 m from a NaPTAN stop
# survived and double-counted (16.7k at 50-75 m, 7.8k at 75-100 m). The NaPTAN
# nearest-neighbour spacing (opposite-side-of-road pairs) has a 166.7 m median
# (p25=98.7 m), so 100 m clears the 50-100 m duplicate spike while leaving a
# genuine opposite-direction stop (~166 m away) in place; 125-150 m would start
# dropping those real opposite-side stops.
BUS_STOP_DEDUP_RADIUS_M = 100.0
def osm_stops_near_naptan(
@ -1652,9 +1681,9 @@ def osm_groceries_colocated_with_geolytix(
An OSM Groceries row is a duplicate when a GEOLYTIX point lies within
``radius_m`` metres AND that point's brand tokens (its ``category``, e.g.
"Tesco", "Co-op") are all present in the OSM row's name — i.e. the same
physical branded store. Brands with no token >= 3 chars (e.g. "M&S") never
match, so they are conservatively kept rather than risk a false drop.
"Tesco", "Co-op", "M&S") are all present in the OSM row's name — i.e. the
same physical branded store. Short brands like "M&S" match via
_GROCERY_SHORT_BRAND_TOKENS; brands that still tokenise to set() are kept.
``osm_groceries`` needs columns ``id``, ``name``, ``lat``, ``lng``;
``geolytix`` needs ``category`` (the brand), ``lat``, ``lng``.
@ -1774,6 +1803,31 @@ def transform(
# pre-deduplicated.
lf = lf.unique(subset=["id", "category"], keep="first", maintain_order=True)
# Relabel OSM Iceland/Farmfoods rows that CATEGORY_MAP put in "Deli &
# Specialty" to the brand, so the genuine stores GEOLYTIX lacks (the grocery
# dedup drops the colocated twins) line up with the GEOLYTIX brand pills and
# the headline grocery metric instead of shadowing them as deli.
for _brand_name, _brand in OSM_BRAND_RELABEL.items():
_is_brand = (
(pl.col("group") == "Groceries")
& (pl.col("category") == "Deli & Specialty")
& (pl.col("name") == _brand_name)
)
lf = lf.with_columns(
pl.when(_is_brand)
.then(pl.lit(_brand))
.otherwise(pl.col("category"))
.alias("category"),
pl.when(_is_brand)
.then(pl.lit(_brand))
.otherwise(pl.col("icon_category"))
.alias("icon_category"),
pl.when(_is_brand)
.then(pl.lit("\U0001f6d2"))
.otherwise(pl.col("emoji"))
.alias("emoji"),
)
naptan_df = pl.scan_parquet(naptan_path).collect()
mask = in_england_mask(
boundary_path,

View file

@ -6,6 +6,29 @@ mod postcodes;
mod property;
pub mod travel_time;
/// Apostrophe-like code points that should be elided (not treated as word breaks) when
/// normalizing search text. Keyboards, autocorrect and copy-paste all produce different glyphs
/// for what users mean as an apostrophe; treating any of them as a separator turns "King's Cross"
/// into the tokens `king s cross` and breaks matching. Covers the straight ASCII apostrophe, the
/// typographic left/right single quotes, grave/acute accents, the modifier-letter apostrophes,
/// prime, and the fullwidth apostrophe.
pub(crate) fn is_apostrophe(ch: char) -> bool {
matches!(
ch,
'\u{0027}' // ' APOSTROPHE
| '\u{0060}' // ` GRAVE ACCENT
| '\u{00B4}' // ´ ACUTE ACCENT
| '\u{02B9}' // ʹ MODIFIER LETTER PRIME
| '\u{02BB}' // ʻ MODIFIER LETTER TURNED COMMA (ʻokina)
| '\u{02BC}' // ʼ MODIFIER LETTER APOSTROPHE
| '\u{2018}' // ' LEFT SINGLE QUOTATION MARK
| '\u{2019}' // ' RIGHT SINGLE QUOTATION MARK
| '\u{201B}' // SINGLE HIGH-REVERSED-9 QUOTATION MARK
| '\u{2032}' // PRIME
| '\u{FF07}' // FULLWIDTH APOSTROPHE
)
}
fn panic_payload_message(payload: &(dyn std::any::Any + Send)) -> String {
if let Some(message) = payload.downcast_ref::<&'static str>() {
(*message).to_string()

View file

@ -161,7 +161,7 @@ pub fn normalize_search_text(text: &str) -> String {
let mut last_was_space = true;
for ch in text.chars() {
if ch == '\'' || ch == '' || ch == '`' {
if super::is_apostrophe(ch) {
continue;
}
@ -769,6 +769,26 @@ impl PlaceData {
mod tests {
use super::*;
#[test]
fn normalize_search_text_elides_every_apostrophe_variant() {
// Whatever glyph the keyboard / autocorrect / paste produced for the apostrophe, the
// normalized form must be identical — otherwise "King's Cross" tokenizes as `king s cross`
// and matching breaks. See is_apostrophe for the full set.
let expected = "kings cross";
for q in [
"King's Cross", // U+0027 straight
"Kings Cross", // U+2019 right single quote
"Kings Cross", // U+2018 left single quote
"King´s Cross", // U+00B4 acute accent
"Kingʼs Cross", // U+02BC modifier letter apostrophe
"Kings Cross", // U+2032 prime
"King`s Cross", // U+0060 grave accent
"Kingʻs Cross", // U+02BB modifier letter turned comma
] {
assert_eq!(normalize_search_text(q), expected, "failed for {q:?}");
}
}
#[test]
fn place_index_tokens_dedup_and_min_length() {
// "a" is too short; aliases split on " | ".

View file

@ -37,7 +37,7 @@ fn tokenize_address_text(text: &str) -> Vec<String> {
for ch in text.chars() {
if ch.is_ascii_alphanumeric() {
current.push(ch.to_ascii_lowercase());
} else if matches!(ch, '\'' | '' | '`') {
} else if crate::data::is_apostrophe(ch) {
continue;
} else if !current.is_empty() {
tokens.push(std::mem::take(&mut current));
@ -784,6 +784,24 @@ impl PropertyData {
mod tests {
use super::*;
#[test]
fn tokenize_address_text_elides_every_apostrophe_variant() {
// The query and the index both run through this tokenizer, so any apostrophe glyph must
// join the word ("O'Brien" -> "obrien") rather than split it; otherwise the same address
// tokenizes differently depending on which quote glyph was typed.
let expected = vec!["obrien".to_string(), "road".to_string()];
for addr in [
"O'Brien Road", // U+0027 straight
"OBrien Road", // U+2019 right single quote
"OBrien Road", // U+2018 left single quote
"O´Brien Road", // U+00B4 acute accent
"OʼBrien Road", // U+02BC modifier letter apostrophe
"OBrien Road", // U+2032 prime
] {
assert_eq!(tokenize_address_text(addr), expected, "failed for {addr:?}");
}
}
#[test]
fn full_postcode_detection_accepts_common_formats() {
assert!(is_full_postcode_compact("SW1A1AA"));

View file

@ -31,10 +31,17 @@ pub struct FeatureConfig {
pub absolute: bool,
}
/// Features whose histogram bins should be exactly 1 unit wide (one per integer).
/// p1/p99 are snapped to integer boundaries before binning.
/// Explicit override for features that should use integer-width histogram bins
/// (one bin per integer, p1/p99 snapped to integer boundaries) even if they
/// don't match the automatic rule in [`has_integer_bins`]. Most integer count
/// features are detected automatically from their config.
pub const INTEGER_BIN_FEATURES: &[&str] = &["Number of bedrooms & living rooms"];
/// Largest p1p99 span (in integer units) for which one-bin-per-integer
/// histograms stay reasonable. Above this, percentile binning is used instead so
/// wide-range features (years, percentages, prices) don't allocate one bin each.
const MAX_INTEGER_BIN_RANGE: f32 = 16.0;
pub struct EnumFeatureConfig {
pub name: &'static str,
/// If set, values are presented in this order instead of alphabetical.
@ -473,11 +480,11 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
high: 98.0,
},
step: 1.0,
description: "Aggregate of serious crime categories per year",
detail: "Sum of violence, robbery, burglary, and weapons possession per year near the postcode, counted from police.uk street-level crime points (anonymised, snapped to nearby map points). This is an area-normalised incident density for the surrounding streets, not a per-resident risk: busy commercial centres rank high however few people live there. Averaged over the months the local police force actually published data; known force gaps (e.g. Greater Manchester since mid-2019) are excluded rather than counted as zero crime.",
description: "Relative density of serious crime categories near the postcode",
detail: "Combined density of violence, robbery, burglary, and weapons possession near the postcode, counted from police.uk street-level crime points (anonymised, snapped to nearby map points). This is an area-normalised incident density for the surrounding streets, not a count of incidents per year and not a per-resident risk: busy commercial centres rank high however few people live there. It is normalised to a median-sized catchment so areas are comparable, and computed over the months the local police force actually published data; known force gaps (e.g. Greater Manchester since mid-2019) are excluded rather than counted as zero crime.",
source: "crime",
prefix: "",
suffix: "/yr",
suffix: "",
raw: false,
absolute: false,
}),
@ -488,11 +495,11 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
high: 98.0,
},
step: 1.0,
description: "Aggregate of minor crime categories per year",
detail: "Sum of anti-social behaviour, shoplifting, bicycle theft, and other lower-severity crime per year near the postcode, counted from police.uk street-level crime points (anonymised, snapped to nearby map points). This is an area-normalised incident density for the surrounding streets, not a per-resident risk: busy commercial centres rank high however few people live there. Averaged over the months the local police force actually published data; known force gaps (e.g. Greater Manchester since mid-2019) are excluded rather than counted as zero crime.",
description: "Relative density of minor crime categories near the postcode",
detail: "Combined density of anti-social behaviour, shoplifting, bicycle theft, and other lower-severity crime near the postcode, counted from police.uk street-level crime points (anonymised, snapped to nearby map points). This is an area-normalised incident density for the surrounding streets, not a count of incidents per year and not a per-resident risk: busy commercial centres rank high however few people live there. It is normalised to a median-sized catchment so areas are comparable, and computed over the months the local police force actually published data; known force gaps (e.g. Greater Manchester since mid-2019) are excluded rather than counted as zero crime.",
source: "crime",
prefix: "",
suffix: "/yr",
suffix: "",
raw: false,
absolute: false,
}),
@ -507,7 +514,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
detail: "Average number of violence and sexual offences per year near the postcode, from police.uk street-level crime data. Includes assault, harassment, and sexual offences.",
source: "crime",
prefix: "",
suffix: "/yr",
suffix: "",
raw: false,
absolute: false,
}),
@ -522,7 +529,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
detail: "Average number of burglary offences per year near the postcode, from police.uk street-level crime data. Includes residential and commercial burglary.",
source: "crime",
prefix: "",
suffix: "/yr",
suffix: "",
raw: false,
absolute: false,
}),
@ -537,7 +544,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
detail: "Average number of robbery offences per year near the postcode, from police.uk street-level crime data. Robbery involves theft with force or threat of force.",
source: "crime",
prefix: "",
suffix: "/yr",
suffix: "",
raw: false,
absolute: false,
}),
@ -552,7 +559,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
detail: "Average number of vehicle crime incidents per year near the postcode, from police.uk street-level crime data. Includes theft of and from vehicles.",
source: "crime",
prefix: "",
suffix: "/yr",
suffix: "",
raw: false,
absolute: false,
}),
@ -567,7 +574,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
detail: "Average number of anti-social behaviour incidents per year near the postcode, from police.uk street-level crime data. Includes nuisance, environmental, and personal anti-social behaviour.",
source: "crime",
prefix: "",
suffix: "/yr",
suffix: "",
raw: false,
absolute: false,
}),
@ -582,7 +589,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
detail: "Average number of criminal damage and arson incidents per year near the postcode, from police.uk street-level crime data.",
source: "crime",
prefix: "",
suffix: "/yr",
suffix: "",
raw: false,
absolute: false,
}),
@ -597,7 +604,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
detail: "Average number of 'other theft' offences per year near the postcode, from police.uk street-level crime data. Includes theft not classified under burglary, vehicle crime, shoplifting, or bicycle theft.",
source: "crime",
prefix: "",
suffix: "/yr",
suffix: "",
raw: false,
absolute: false,
}),
@ -612,7 +619,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
detail: "Average number of theft from the person offences per year near the postcode, from police.uk street-level crime data. Includes pickpocketing and bag snatching without force.",
source: "crime",
prefix: "",
suffix: "/yr",
suffix: "",
raw: false,
absolute: false,
}),
@ -627,7 +634,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
detail: "Average number of shoplifting offences per year near the postcode, from police.uk street-level crime data.",
source: "crime",
prefix: "",
suffix: "/yr",
suffix: "",
raw: false,
absolute: false,
}),
@ -642,7 +649,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
detail: "Average number of bicycle theft offences per year near the postcode, from police.uk street-level crime data.",
source: "crime",
prefix: "",
suffix: "/yr",
suffix: "",
raw: false,
absolute: false,
}),
@ -657,7 +664,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
detail: "Average number of drug offences per year near the postcode, from police.uk street-level crime data. Includes possession and trafficking offences.",
source: "crime",
prefix: "",
suffix: "/yr",
suffix: "",
raw: false,
absolute: false,
}),
@ -672,7 +679,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
detail: "Average number of possession of weapons offences per year near the postcode, from police.uk street-level crime data.",
source: "crime",
prefix: "",
suffix: "/yr",
suffix: "",
raw: false,
absolute: false,
}),
@ -687,7 +694,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
detail: "Average number of public order offences per year near the postcode, from police.uk street-level crime data. Includes causing fear, alarm, or distress.",
source: "crime",
prefix: "",
suffix: "/yr",
suffix: "",
raw: false,
absolute: false,
}),
@ -702,7 +709,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
detail: "Average number of other crime offences per year near the postcode, from police.uk street-level crime data. A catch-all category for offences not classified elsewhere.",
source: "crime",
prefix: "",
suffix: "/yr",
suffix: "",
raw: false,
absolute: false,
}),
@ -734,7 +741,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
},
step: 1.0,
description: "Percentage of population identifying as White",
detail: "From the 2021 Census. Percentage of the local authority population identifying as White (English, Welsh, Scottish, Northern Irish, British, Irish, Gypsy or Irish Traveller, Roma, or any other White background).",
detail: "From the 2021 Census. Percentage of the local neighbourhood (LSOA) population identifying as White (English, Welsh, Scottish, Northern Irish, British, Irish, Gypsy or Irish Traveller, Roma, or any other White background).",
source: "ethnicity",
prefix: "",
suffix: "%",
@ -749,7 +756,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
},
step: 0.1,
description: "Percentage of population identifying as South Asian",
detail: "From the 2021 Census. Percentage of the local authority population identifying as Indian, Pakistani, Bangladeshi, or any other Asian background.",
detail: "From the 2021 Census. Percentage of the local neighbourhood (LSOA) population identifying as Indian, Pakistani, or Bangladeshi.",
source: "ethnicity",
prefix: "",
suffix: "%",
@ -764,7 +771,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
},
step: 0.1,
description: "Percentage of population identifying as Black",
detail: "From the 2021 Census. Percentage of the local authority population identifying as Black, Black British, Caribbean, or African.",
detail: "From the 2021 Census. Percentage of the local neighbourhood (LSOA) population identifying as Black, Black British, Caribbean, or African.",
source: "ethnicity",
prefix: "",
suffix: "%",
@ -779,7 +786,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
},
step: 0.1,
description: "Percentage of population identifying as East Asian",
detail: "From the 2021 Census. Percentage of the local authority population identifying as Chinese.",
detail: "From the 2021 Census. Percentage of the local neighbourhood (LSOA) population identifying as Chinese.",
source: "ethnicity",
prefix: "",
suffix: "%",
@ -794,7 +801,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
},
step: 0.1,
description: "Percentage of population identifying as Southeast Asian",
detail: "From the 2021 Census. Percentage of the local authority population identifying as another (non-Chinese) East or Southeast Asian background, e.g. Vietnamese, Filipino, or Thai.",
detail: "From the 2021 Census. Percentage of the local neighbourhood (LSOA) population identifying as another (non-Chinese) East or Southeast Asian background, e.g. Vietnamese, Filipino, or Thai.",
source: "ethnicity",
prefix: "",
suffix: "%",
@ -809,7 +816,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
},
step: 0.1,
description: "Percentage of population identifying as Mixed or Multiple ethnic groups",
detail: "From the 2021 Census. Percentage of the local authority population identifying as Mixed or Multiple ethnic groups (White and Black Caribbean, White and Black African, White and Asian, or any other Mixed or Multiple background).",
detail: "From the 2021 Census. Percentage of the local neighbourhood (LSOA) population identifying as Mixed or Multiple ethnic groups (White and Black Caribbean, White and Black African, White and Asian, or any other Mixed or Multiple background).",
source: "ethnicity",
prefix: "",
suffix: "%",
@ -824,7 +831,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
},
step: 0.1,
description: "Percentage of population identifying as Other ethnic group",
detail: "From the 2021 Census. Percentage of the local authority population identifying as Other ethnic group (Arab or any other ethnic group not covered by the main categories).",
detail: "From the 2021 Census. Percentage of the local neighbourhood (LSOA) population identifying as Other ethnic group (Arab or any other ethnic group not covered by the main categories).",
source: "ethnicity",
prefix: "",
suffix: "%",
@ -997,9 +1004,48 @@ pub fn order_for(name: &str) -> Option<&'static [&'static str]> {
.flatten()
}
/// Whether this feature should use integer-width histogram bins.
/// Whether this feature should use integer-width histogram bins: one bin per
/// integer value with p1/p99 snapped to integer boundaries. This keeps the
/// histogram bars aligned 1:1 with the integer axis labels in the UI (otherwise
/// fractional percentiles and non-unit bin widths make the lit bar drift away
/// from its label, e.g. a value of 3 landing under the "2" tick).
///
/// Applies to small-range integer-valued count features (school catchments,
/// bedrooms, …) plus the dynamic POI count features. Wide-range or fractional
/// features (years, percentages, crime averages) keep percentile binning.
pub fn has_integer_bins(name: &str) -> bool {
INTEGER_BIN_FEATURES.contains(&name) || dynamic_poi_count_radius(name).is_some()
if INTEGER_BIN_FEATURES.contains(&name) || dynamic_poi_count_radius(name).is_some() {
return true;
}
matches!(numeric_config_for(name), Some(c) if is_integer_count_feature(c))
}
/// True for a numeric feature whose values are small integer counts: unit step
/// with fixed integer bounds spanning a small enough range for one bin per value.
fn is_integer_count_feature(config: &FeatureConfig) -> bool {
if config.step != 1.0 {
return false;
}
match config.bounds {
Bounds::Fixed { min, max } => {
min.fract() == 0.0
&& max.fract() == 0.0
&& max > min
&& (max - min) <= MAX_INTEGER_BIN_RANGE
}
Bounds::Percentile { .. } => false,
}
}
/// Look up the static numeric feature config by exact name.
fn numeric_config_for(name: &str) -> Option<&'static FeatureConfig> {
FEATURE_GROUPS
.iter()
.flat_map(|group| group.features.iter())
.find_map(|feature| match feature {
Feature::Numeric(c) if c.name == name => Some(c),
_ => None,
})
}
/// Look up the Bounds config for a numeric feature by name.
@ -1084,3 +1130,44 @@ pub const POI_GROUP_ORDER: &[&str] = &[
"Services",
"Shops",
];
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn integer_bins_cover_small_integer_count_features() {
// Small integer count features must get one bin per integer so the
// histogram bars line up with their integer axis labels.
for name in [
"Good+ primary school catchments",
"Good+ secondary school catchments",
"Outstanding primary school catchments",
"Outstanding secondary school catchments",
"Number of bedrooms & living rooms",
] {
assert!(has_integer_bins(name), "{name} should use integer bins");
}
}
#[test]
fn integer_bins_exclude_wide_range_and_fractional_features() {
// step == 1.0 but wide-range (would explode bin count) or fractional
// (averages/percentiles) features must keep percentile binning.
for name in [
"Construction year", // Fixed but spans ~2000 years
"Date of last transaction", // Fixed, fractional years
"Income Score", // 0..100 percentile
"% White", // 0..100 percentage
"Noise (dB)", // 50..80, range > threshold
"Serious crime (avg/yr)", // Percentile bounds, fractional
"Interior height (m)", // step 0.1
"Estimated current price", // step 10000
] {
assert!(
!has_integer_bins(name),
"{name} should not use integer bins"
);
}
}
}

View file

@ -440,6 +440,11 @@ pub async fn get_export(
.map_err(|err| (StatusCode::BAD_REQUEST, err).into_response())?
};
let has_poi_filters = !parsed_poi_filters.is_empty();
// The active filters decide which feature columns land on the "Selected"
// sheet. We keep them for column selection in both modes; they are only
// *applied* as row filters in bounds mode (in list mode the user picks the
// postcodes explicitly, so rows are never filtered).
let column_filters_str = params.filters.clone();
let filters_str = if is_postcode_mode {
None
} else {
@ -653,7 +658,7 @@ pub async fn get_export(
};
// Determine column order: filter features first, then remaining
let filter_feature_names = extract_filter_feature_names(filters_str.as_deref());
let filter_feature_names = extract_filter_feature_names(column_filters_str.as_deref());
let field_indices = parse_field_indices_with_poi(
fields_str.as_deref(),
@ -841,18 +846,16 @@ pub async fn get_export(
frontend_params
);
// Bounds mode: two sheets — "Selected" (filter features with link + screenshot)
// and "All Data" (all features).
// List mode: single sheet "Postcodes" with all data, no link or screenshot
// (the supplied list isn't tied to a map view).
let sheet_configs: Vec<(&str, &[usize], bool)> = if postcode_list_entries.is_some() {
vec![("Postcodes", &all_feature_indices, false)]
} else {
vec![
("Selected", &filter_feature_indices, true),
("All Data", &all_feature_indices, false),
]
};
// Two sheets in both modes: "Selected" (just the features behind the
// active filters) and "All Data" (every feature). The Selected sheet
// carries the dashboard link + screenshot only in bounds mode, where the
// export is tied to a map view; in list mode the user picked the
// postcodes explicitly, so there's no map view to link or screenshot.
let is_list_mode = postcode_list_entries.is_some();
let sheet_configs: Vec<(&str, &[usize], bool)> = vec![
("Selected", &filter_feature_indices, !is_list_mode),
("All Data", &all_feature_indices, false),
];
for (sheet_name, feat_indices, include_header) in &sheet_configs {
let sheet = workbook.add_worksheet();

View file

@ -368,12 +368,7 @@ pub fn compute_crime_by_year(
/// force-level publication gaps (e.g. Greater Manchester ends mid-2019) as
/// stale data instead of letting old numbers read as current.
pub fn crime_latest_available_year(crime_by_year: &CrimeByYearData) -> Option<i32> {
crime_by_year
.years_by_type
.iter()
.flatten()
.copied()
.max()
crime_by_year.years_by_type.iter().flatten().copied().max()
}
pub fn compute_poi_feature_stats(