Changes
This commit is contained in:
parent
3a3f899ea2
commit
128b3191e7
68 changed files with 28060 additions and 1152 deletions
14
CLAUDE.md
14
CLAUDE.md
|
|
@ -102,13 +102,13 @@ Rust + Axum. Loads parquet into memory at startup.
|
||||||
- `GET /api/pois?bounds=&categories=` — POIs by bounds (max 5000)
|
- `GET /api/pois?bounds=&categories=` — POIs by bounds (max 5000)
|
||||||
- `GET /api/poi-categories` — Available POI category names
|
- `GET /api/poi-categories` — Available POI category names
|
||||||
|
|
||||||
Serves `frontend/dist/` as static fallback in production.
|
Serves `frontend/dist/` as static fallback in production **only** when `--dist` is explicitly provided. When `--dist` is set, the server panics at startup if `index.html` is unreadable. When omitted (dev mode), static serving and OG injection are disabled.
|
||||||
|
|
||||||
**Data representation (unified model):**
|
**Data representation (unified model):**
|
||||||
- All features (numeric and enum): row-major flat `Vec<f32>`, NaN = null
|
- All features (numeric and enum): row-major flat `Vec<f32>`, NaN = null
|
||||||
- Enum features: stored as f32 indices (0.0, 1.0, 2.0...) with `enum_values: FxHashMap<usize, Vec<String>>` mapping feature index → string values
|
- Enum features: stored as f32 indices (0.0, 1.0, 2.0...) with `enum_values: FxHashMap<usize, Vec<String>>` mapping feature index → string values
|
||||||
- String fields (address, postcode): interned/packed for memory efficiency
|
- String fields (address, postcode): interned/packed for memory efficiency
|
||||||
- The server accepts the parquet path as a CLI argument (defaults to `data_sources/processed/wide.parquet`)
|
- All CLI args are required (no hidden defaults). Optional services use `Option<String>`: `r5_url` (travel time disabled when None), `pocketbase_admin_email`/`password` (collection auto-creation skipped when None). Required config like `ollama_model` and `public_url` must be explicitly provided via env or CLI.
|
||||||
|
|
||||||
### Frontend (`frontend/`)
|
### Frontend (`frontend/`)
|
||||||
|
|
||||||
|
|
@ -127,7 +127,7 @@ React 18 + TypeScript. deck.gl `H3HexagonLayer` over MapLibre GL. TailwindCSS. N
|
||||||
- `useUrlSync` — URL state synchronization
|
- `useUrlSync` — URL state synchronization
|
||||||
|
|
||||||
**Key patterns:**
|
**Key patterns:**
|
||||||
- URL encodes view/filters/POI categories/active tab as query params for shareable links. Only the current format is supported — no legacy parameter parsing (old `v=`, `f=`, or tab abbreviations are not handled).
|
- URL encodes view/filters/POI categories/active tab as query params for shareable links. Only the current format is supported — no legacy parameter parsing (old `v=`, `f=`, or tab abbreviations are not handled). `tmode` is always serialized when travel time is active (no implicit default); parsing throws if `tmode` is missing when `dest` is present.
|
||||||
- AbortControllers cancel in-flight requests on new queries (150ms debounce)
|
- AbortControllers cancel in-flight requests on new queries (150ms debounce)
|
||||||
- Zoom → H3 resolution defined in `consts.ts` `ZOOM_TO_RESOLUTION_THRESHOLDS`: `<7.5→5, <9.5→6, <10.5→8, <12→9, ≥12→10`
|
- Zoom → H3 resolution defined in `consts.ts` `ZOOM_TO_RESOLUTION_THRESHOLDS`: `<7.5→5, <9.5→6, <10.5→8, <12→9, ≥12→10`
|
||||||
- `POSTCODE_ZOOM_THRESHOLD = 15`: below 15 shows H3 hexagons, at/above 15 shows postcode polygons
|
- `POSTCODE_ZOOM_THRESHOLD = 15`: below 15 shows H3 hexagons, at/above 15 shows postcode polygons
|
||||||
|
|
@ -271,7 +271,12 @@ Every UI element must use the correct token from this table. Do not invent new p
|
||||||
|
|
||||||
## Coding Preferences
|
## Coding Preferences
|
||||||
|
|
||||||
- **No backwards compatibility, no silent fallbacks**: Never add fallback codepaths for old data formats, legacy URL parameters, or alternate field names. Never silently swallow errors — always error loudly (return an error, panic, or at minimum log). If something is wrong, the code should fail visibly. One canonical name per field, one format per API, one way to do things.
|
- **No backwards compatibility, no silent fallbacks**: Never add fallback codepaths for old data formats, legacy URL parameters, or alternate field names. Never silently swallow errors — always error loudly (return an error, panic, or at minimum log). If something is wrong, the code should fail visibly. One canonical name per field, one format per API, one way to do things. Specific patterns:
|
||||||
|
- Use `Option<String>` for truly optional config, never `default_value = ""` with `.is_empty()` checks
|
||||||
|
- Use `expect()` not `unwrap_or(0.0)` when a value is logically guaranteed to be present
|
||||||
|
- Return error responses on upstream failures, never silently drop results
|
||||||
|
- Don't add `#[serde(default)]` on `Option<T>` fields — serde already defaults them to `None`
|
||||||
|
- Required query params should be non-Option types so serde rejects missing params with 400 automatically
|
||||||
- **Unified data models over special-casing**: Prefer storing different data types uniformly (e.g., enums as f32 indices alongside numeric features) rather than maintaining separate code paths
|
- **Unified data models over special-casing**: Prefer storing different data types uniformly (e.g., enums as f32 indices alongside numeric features) rather than maintaining separate code paths
|
||||||
- **Terse tests**: Test what matters in as few tests as possible — don't overcomplicate with excessive setup or edge cases that don't add value
|
- **Terse tests**: Test what matters in as few tests as possible — don't overcomplicate with excessive setup or edge cases that don't add value
|
||||||
- **Extract and organize**: Group related utilities into proper modules (e.g., `utils/`, `parsing/`) rather than leaving helpers scattered
|
- **Extract and organize**: Group related utilities into proper modules (e.g., `utils/`, `parsing/`) rather than leaving helpers scattered
|
||||||
|
|
@ -316,6 +321,7 @@ Follow these conventions in all Rust code:
|
||||||
- **Fuzzy join**: Groups by postcode, uses `thefuzz.token_sort_ratio` with numeric token compatibility, greedy assignment from highest score
|
- **Fuzzy join**: Groups by postcode, uses `thefuzz.token_sort_ratio` with numeric token compatibility, greedy assignment from highest score
|
||||||
- **Filter parsing is strict**: `parse_filters()` returns `Result` — malformed entries, unknown feature names, and unparseable numbers all return 400 Bad Request. No silent skipping of invalid filters.
|
- **Filter parsing is strict**: `parse_filters()` returns `Result` — malformed entries, unknown feature names, and unparseable numbers all return 400 Bad Request. No silent skipping of invalid filters.
|
||||||
- **Data loading is strict**: `extract_string_col` and `lookup_enum_value` take a single column name (no fallback names). H3 precomputation panics on invalid coordinates. Required parquet columns must exist at startup.
|
- **Data loading is strict**: `extract_string_col` and `lookup_enum_value` take a single column name (no fallback names). H3 precomputation panics on invalid coordinates. Required parquet columns must exist at startup.
|
||||||
|
- **Travel time is strict**: `mode` param is required (400) when `destination` is set — no silent default to "car". R5 failures return 502 Bad Gateway, not silent omission. `r5_url` is `Option<String>` — returns 503 if travel time requested without R5 configured.
|
||||||
- **Filter bounds format**: `south,west,north,east` (not standard bbox order)
|
- **Filter bounds format**: `south,west,north,east` (not standard bbox order)
|
||||||
- **Server-side AABB filtering**: Both `/api/hexagons` and `/api/postcodes` filter results by bounding-box intersection with query bounds. Hexagons use `h3_cell_bounds()` (h3o returns degrees, not radians). Postcodes compute polygon AABB from vertices. See `bounds_intersect()` in `parsing/bounds.rs`.
|
- **Server-side AABB filtering**: Both `/api/hexagons` and `/api/postcodes` filter results by bounding-box intersection with query bounds. Hexagons use `h3_cell_bounds()` (h3o returns degrees, not radians). Postcodes compute polygon AABB from vertices. See `bounds_intersect()` in `parsing/bounds.rs`.
|
||||||
- **GridIndex returns slightly more than requested**: The 0.01° grid cells mean properties up to ~1km outside the viewport may be returned. The AABB filter in the route handlers catches these extras.
|
- **GridIndex returns slightly more than requested**: The 0.01° grid cells mean properties up to ~1km outside the viewport may be returned. The AABB filter in the route handlers catches these extras.
|
||||||
|
|
|
||||||
|
|
@ -15,7 +15,7 @@ RUN cargo build --release
|
||||||
|
|
||||||
# Stage 3: Runtime
|
# Stage 3: Runtime
|
||||||
FROM debian:bookworm-slim
|
FROM debian:bookworm-slim
|
||||||
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates && rm -rf /var/lib/apt/lists/*
|
RUN apt-get update && apt-get install -y --no-install-recommends ca-certificates curl && rm -rf /var/lib/apt/lists/*
|
||||||
WORKDIR /app
|
WORKDIR /app
|
||||||
COPY --from=server /app/server-rs/target/release/property-map-server ./
|
COPY --from=server /app/server-rs/target/release/property-map-server ./
|
||||||
COPY --from=frontend /app/frontend/dist ./frontend/dist/
|
COPY --from=frontend /app/frontend/dist ./frontend/dist/
|
||||||
|
|
@ -27,5 +27,7 @@ COPY property-data/uk.pmtiles ./data/
|
||||||
COPY manual-data/postcode_boundaries ./data/postcode_boundaries/
|
COPY manual-data/postcode_boundaries ./data/postcode_boundaries/
|
||||||
|
|
||||||
EXPOSE 8001
|
EXPOSE 8001
|
||||||
|
HEALTHCHECK --interval=30s --timeout=5s --start-period=120s --retries=3 \
|
||||||
|
CMD curl -f http://localhost:8001/health || exit 1
|
||||||
ENTRYPOINT ["./property-map-server"]
|
ENTRYPOINT ["./property-map-server"]
|
||||||
CMD ["--data", "/app/data/wide.parquet", "--pois", "/app/data/filtered_uk_pois.parquet", "--places", "/app/data/places.parquet", "--tiles", "/app/data/uk.pmtiles", "--postcodes", "/app/data/postcode_boundaries"]
|
CMD ["--data", "/app/data/wide.parquet", "--pois", "/app/data/filtered_uk_pois.parquet", "--places", "/app/data/places.parquet", "--tiles", "/app/data/uk.pmtiles", "--postcodes", "/app/data/postcode_boundaries", "--dist", "/app/frontend/dist"]
|
||||||
|
|
|
||||||
|
|
@ -24,6 +24,8 @@ POI_PROXIMITY := $(DATA_DIR)/poi_proximity.parquet
|
||||||
EPC_PP := $(DATA_DIR)/epc_pp.parquet
|
EPC_PP := $(DATA_DIR)/epc_pp.parquet
|
||||||
WIDE := $(DATA_DIR)/wide.parquet
|
WIDE := $(DATA_DIR)/wide.parquet
|
||||||
PRICE_INDEX := $(DATA_DIR)/price_index.parquet
|
PRICE_INDEX := $(DATA_DIR)/price_index.parquet
|
||||||
|
RENO_PREMIUM := $(DATA_DIR)/renovation_premium.parquet
|
||||||
|
HEDONIC_MODEL := $(DATA_DIR)/hedonic_model.json
|
||||||
PRICES_STAMP := $(DATA_DIR)/.prices_done
|
PRICES_STAMP := $(DATA_DIR)/.prices_done
|
||||||
EPC := $(MANUAL_DATA)/certificates.csv
|
EPC := $(MANUAL_DATA)/certificates.csv
|
||||||
JT_BANK := $(MANUAL_DATA)/journey_times_bank.parquet
|
JT_BANK := $(MANUAL_DATA)/journey_times_bank.parquet
|
||||||
|
|
@ -263,6 +265,13 @@ $(WIDE): $(EPC_PP) $(ARCGIS) $(IOD) $(POI_PROXIMITY) $(JT_BANK) $(JT_FITZROVIA)
|
||||||
$(PRICE_INDEX): $(WIDE)
|
$(PRICE_INDEX): $(WIDE)
|
||||||
uv run python -m pipeline.transform.price_index --input $(WIDE) --output $@
|
uv run python -m pipeline.transform.price_index --input $(WIDE) --output $@
|
||||||
|
|
||||||
$(PRICES_STAMP): $(WIDE) $(PRICE_INDEX)
|
$(RENO_PREMIUM): $(WIDE) $(PRICE_INDEX)
|
||||||
uv run python -m pipeline.transform.price_estimate --input $(WIDE) --index $(PRICE_INDEX)
|
uv run python -m pipeline.transform.renovation_premium --input $(WIDE) --index $(PRICE_INDEX) --output $@
|
||||||
|
|
||||||
|
$(HEDONIC_MODEL): $(WIDE)
|
||||||
|
uv run python -m pipeline.transform.hedonic_quality --input $(WIDE) --output $@
|
||||||
|
|
||||||
|
$(PRICES_STAMP): $(WIDE) $(PRICE_INDEX) $(RENO_PREMIUM) $(HEDONIC_MODEL)
|
||||||
|
uv run python -m pipeline.transform.price_estimate --input $(WIDE) --index $(PRICE_INDEX) \
|
||||||
|
--renovation-premium $(RENO_PREMIUM) --hedonic-model $(HEDONIC_MODEL)
|
||||||
@touch $@
|
@touch $@
|
||||||
|
|
|
||||||
11
README.md
11
README.md
|
|
@ -34,9 +34,6 @@ rm data/d29f0314840ef7dcbb5cde66e383fe08059dab5a.zip
|
||||||
https://xploria.co.uk/data-sources/
|
https://xploria.co.uk/data-sources/
|
||||||
|
|
||||||
|
|
||||||
epc oopt out
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
We all care about different things in our homes and living environments. Some of us are weary of noise and would like to avoid living next to a loud airfield as much as possible. And some of us are avid plane spotters.
|
We all care about different things in our homes and living environments. Some of us are weary of noise and would like to avoid living next to a loud airfield as much as possible. And some of us are avid plane spotters.
|
||||||
|
|
@ -77,17 +74,9 @@ make -f Makefile.data tiles
|
||||||
Add licensing to the app. By default, anonymous users can use the map but only in central london. if they try zooming out, the server refuses to provide data and the users will be prompted to buy a lifetime license to continue (or zoom back in). Just before buying a license, they have to register by providing their email address and password, then they need to complete the stripe check out workflow. Implement the full pocketbase/server/frontend integration. For admins, give an option to generate an invite link, opening which prompts you to register and gives you a free license forever. Have a cool animation with party poppers on the successful acquiring of a license. For non-admin users, allow inviting friends for 30% off the price. Also add a support page that shows my email address, and add a FAQ on the same page too.
|
Add licensing to the app. By default, anonymous users can use the map but only in central london. if they try zooming out, the server refuses to provide data and the users will be prompted to buy a lifetime license to continue (or zoom back in). Just before buying a license, they have to register by providing their email address and password, then they need to complete the stripe check out workflow. Implement the full pocketbase/server/frontend integration. For admins, give an option to generate an invite link, opening which prompts you to register and gives you a free license forever. Have a cool animation with party poppers on the successful acquiring of a license. For non-admin users, allow inviting friends for 30% off the price. Also add a support page that shows my email address, and add a FAQ on the same page too.
|
||||||
|
|
||||||
-
|
-
|
||||||
- the area stastics are missing for postcodes, they only work for hexagons
|
|
||||||
- add blue/green rollout
|
- add blue/green rollout
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
Stop wrapping everything in cards. Be bold and stop being lazy around text formatting.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
uv run python scripts/remove_bg.py house-og.png 200 house.png
|
uv run python scripts/remove_bg.py house-og.png 200 house.png
|
||||||
|
|
|
||||||
|
|
@ -19,7 +19,7 @@ tasks:
|
||||||
download:places:
|
download:places:
|
||||||
desc: Extract place names from OSM PBF
|
desc: Extract place names from OSM PBF
|
||||||
cmds:
|
cmds:
|
||||||
- uv run python -m pipeline.download.places --output ./property_data/places.parquet {{.CLI_ARGS}}
|
- uv run python -m pipeline.download.places --output ./property-data/places.parquet {{.CLI_ARGS}}
|
||||||
|
|
||||||
test:
|
test:
|
||||||
desc: Run all tests (Python and Rust)
|
desc: Run all tests (Python and Rust)
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,7 @@
|
||||||
|
x-credentials:
|
||||||
|
pb-email: &pb-email admin@propertymap.local
|
||||||
|
pb-password: &pb-password propertymap-dev-2024
|
||||||
|
|
||||||
services:
|
services:
|
||||||
server:
|
server:
|
||||||
image: rust:1.84
|
image: rust:1.84
|
||||||
|
|
@ -21,11 +25,14 @@ services:
|
||||||
- ./property-data:/app/data:ro
|
- ./property-data:/app/data:ro
|
||||||
environment:
|
environment:
|
||||||
POCKETBASE_URL: http://pocketbase:8090
|
POCKETBASE_URL: http://pocketbase:8090
|
||||||
POCKETBASE_ADMIN_EMAIL: ${POCKETBASE_ADMIN_EMAIL:-}
|
POCKETBASE_ADMIN_EMAIL: *pb-email
|
||||||
POCKETBASE_ADMIN_PASSWORD: ${POCKETBASE_ADMIN_PASSWORD:-}
|
POCKETBASE_ADMIN_PASSWORD: *pb-password
|
||||||
SCREENSHOT_URL: http://screenshot:8002
|
SCREENSHOT_URL: http://screenshot:8002
|
||||||
OLLAMA_URL: http://host.docker.internal:11434
|
OLLAMA_URL: http://host.docker.internal:11434
|
||||||
|
OLLAMA_MODEL: gpt-oss:20b
|
||||||
|
PUBLIC_URL: https://perfectpostcodes.schmelczer.dev
|
||||||
R5_URL: http://r5:8003
|
R5_URL: http://r5:8003
|
||||||
|
GOOGLE_MAPS_API_KEY: "AIzaSyBgBn9LjrxHCjb9j1LZbLYpEdCJj-NkHPY"
|
||||||
depends_on:
|
depends_on:
|
||||||
pocketbase:
|
pocketbase:
|
||||||
condition: service_healthy
|
condition: service_healthy
|
||||||
|
|
@ -83,6 +90,9 @@ services:
|
||||||
- pb-data:/pb/pb_data
|
- pb-data:/pb/pb_data
|
||||||
networks:
|
networks:
|
||||||
- dev-network
|
- dev-network
|
||||||
|
environment:
|
||||||
|
PB_ADMIN_EMAIL: *pb-email
|
||||||
|
PB_ADMIN_PASSWORD: *pb-password
|
||||||
healthcheck:
|
healthcheck:
|
||||||
test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:8090/api/health"]
|
test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:8090/api/health"]
|
||||||
interval: 10s
|
interval: 10s
|
||||||
|
|
@ -90,6 +100,47 @@ services:
|
||||||
retries: 3
|
retries: 3
|
||||||
start_period: 5s
|
start_period: 5s
|
||||||
|
|
||||||
|
gluetun:
|
||||||
|
image: qmcgaw/gluetun:v3.40.4
|
||||||
|
volumes:
|
||||||
|
- gluetun-cache-v2:/gluetun
|
||||||
|
- gluetun-auth:/gluetun/auth:ro
|
||||||
|
environment:
|
||||||
|
# See https://github.com/qdm12/gluetun-wiki/tree/main/setup#setup
|
||||||
|
VPN_SERVICE_PROVIDER: mullvad
|
||||||
|
VPN_TYPE: wireguard
|
||||||
|
WIREGUARD_PRIVATE_KEY: "8FFKmtTvDsZlShnKl/opDDwCwb9v2ox4+Kkl3wX+9Gw="
|
||||||
|
WIREGUARD_ADDRESSES: "10.66.109.86/32"
|
||||||
|
OWNED_ONLY: "yes"
|
||||||
|
UPDATER_PERIOD: 24h
|
||||||
|
SERVER_COUNTRIES: Serbia,Slovakia,Croatia,Austria,Denmark,Finland
|
||||||
|
TZ: $TIME_ZONE
|
||||||
|
restart: unless-stopped
|
||||||
|
ports:
|
||||||
|
- "1234:1234"
|
||||||
|
healthcheck:
|
||||||
|
test: "wget -q https://www.google.com || exit 1"
|
||||||
|
interval: 1m
|
||||||
|
timeout: 15s
|
||||||
|
retries: 2
|
||||||
|
cap_add:
|
||||||
|
- NET_ADMIN
|
||||||
|
devices:
|
||||||
|
- /dev/net/tun:/dev/net/tun
|
||||||
|
|
||||||
|
|
||||||
|
finder:
|
||||||
|
build: ./finder
|
||||||
|
init: true
|
||||||
|
network_mode: service:gluetun
|
||||||
|
volumes:
|
||||||
|
- ./finder:/app
|
||||||
|
- ./property-data/arcgis_data.parquet:/data/arcgis_data.parquet:ro
|
||||||
|
depends_on:
|
||||||
|
gluetun:
|
||||||
|
condition: service_healthy
|
||||||
|
restart: unless-stopped
|
||||||
|
|
||||||
r5:
|
r5:
|
||||||
init: true
|
init: true
|
||||||
build: ./r5-java
|
build: ./r5-java
|
||||||
|
|
@ -119,6 +170,8 @@ volumes:
|
||||||
frontend-node-modules:
|
frontend-node-modules:
|
||||||
screenshot-cache:
|
screenshot-cache:
|
||||||
r5-network:
|
r5-network:
|
||||||
|
gluetun-cache-v2:
|
||||||
|
gluetun-auth:
|
||||||
|
|
||||||
networks:
|
networks:
|
||||||
dev-network:
|
dev-network:
|
||||||
|
|
|
||||||
11
finder/Dockerfile
Normal file
11
finder/Dockerfile
Normal file
|
|
@ -0,0 +1,11 @@
|
||||||
|
FROM python:3.12-slim
|
||||||
|
|
||||||
|
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
|
||||||
|
|
||||||
|
WORKDIR /app
|
||||||
|
COPY pyproject.toml ./
|
||||||
|
RUN uv pip install --system -r pyproject.toml
|
||||||
|
|
||||||
|
COPY main.py ./
|
||||||
|
|
||||||
|
CMD ["python3", "main.py"]
|
||||||
710
finder/main.py
Normal file
710
finder/main.py
Normal file
|
|
@ -0,0 +1,710 @@
|
||||||
|
import logging
|
||||||
|
import math
|
||||||
|
import os
|
||||||
|
import random
|
||||||
|
import re
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
from collections import defaultdict
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
import polars as pl
|
||||||
|
from flask import Flask, jsonify, send_from_directory
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Logging
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
LOG_DIR = Path("/app/data")
|
||||||
|
LOG_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
|
||||||
|
logging.basicConfig(
|
||||||
|
level=logging.DEBUG,
|
||||||
|
format="%(asctime)s [%(levelname)s] %(message)s",
|
||||||
|
handlers=[
|
||||||
|
logging.StreamHandler(),
|
||||||
|
logging.FileHandler(LOG_DIR / "rightmove.log"),
|
||||||
|
],
|
||||||
|
)
|
||||||
|
log = logging.getLogger("rightmove")
|
||||||
|
log.setLevel(logging.DEBUG)
|
||||||
|
logging.getLogger("httpx").setLevel(logging.WARNING)
|
||||||
|
logging.getLogger("httpcore").setLevel(logging.WARNING)
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Constants
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
ARCGIS_PATH = os.environ.get("ARCGIS_PATH", "/data/arcgis_data.parquet")
|
||||||
|
DATA_DIR = Path("/app/data")
|
||||||
|
PAGE_SIZE = 24
|
||||||
|
MAX_PAGES_PER_OUTCODE = 42 # 24*42 = 1008, safety cap per outcode
|
||||||
|
DELAY_BETWEEN_PAGES = 1.0
|
||||||
|
DELAY_BETWEEN_OUTCODES = 2.0
|
||||||
|
MAX_RETRIES = 3
|
||||||
|
RETRY_BASE_DELAY = 2.0
|
||||||
|
GRID_CELL_SIZE = 0.01 # degrees for postcode spatial index
|
||||||
|
SEED = 42
|
||||||
|
|
||||||
|
TYPEAHEAD_URL = "https://los.rightmove.co.uk/typeahead"
|
||||||
|
SEARCH_URL = "https://www.rightmove.co.uk/api/property-search/listing/search"
|
||||||
|
RIGHTMOVE_BASE = "https://www.rightmove.co.uk"
|
||||||
|
|
||||||
|
PROPERTY_TYPE_MAP = {
|
||||||
|
"Detached": "Detached",
|
||||||
|
"Semi-Detached": "Semi-Detached",
|
||||||
|
"Terraced": "Terraced",
|
||||||
|
"End of Terrace": "Terraced",
|
||||||
|
"Mid Terrace": "Terraced",
|
||||||
|
"Flat": "Flat",
|
||||||
|
"Maisonette": "Flat",
|
||||||
|
"Studio": "Flat",
|
||||||
|
"Apartment": "Flat",
|
||||||
|
"Penthouse": "Flat",
|
||||||
|
"Ground Flat": "Flat",
|
||||||
|
"Detached Bungalow": "Detached",
|
||||||
|
"Semi-Detached Bungalow": "Semi-Detached",
|
||||||
|
"Town House": "Terraced",
|
||||||
|
"Link Detached": "Detached",
|
||||||
|
"Link Detached House": "Detached",
|
||||||
|
"Bungalow": "Other",
|
||||||
|
"Cottage": "Other",
|
||||||
|
"Park Home": "Other",
|
||||||
|
"Land": "Other",
|
||||||
|
"Farm / Barn": "Other",
|
||||||
|
"House": "Detached",
|
||||||
|
"Not Specified": "Other",
|
||||||
|
"Chalet": "Other",
|
||||||
|
"Barn Conversion": "Other",
|
||||||
|
"Coach House": "Other",
|
||||||
|
"Character Property": "Other",
|
||||||
|
"Cluster House": "Other",
|
||||||
|
"Retirement Property": "Flat",
|
||||||
|
"Plot": "Other",
|
||||||
|
"Garages": "Other",
|
||||||
|
"Mews": "Terraced",
|
||||||
|
}
|
||||||
|
|
||||||
|
CHANNELS = [
|
||||||
|
{"channel": "BUY", "transactionType": "BUY", "sortType": "2"},
|
||||||
|
{"channel": "RENT", "transactionType": "LETTING", "sortType": "6"},
|
||||||
|
]
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Postcode spatial index
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
class PostcodeSpatialIndex:
|
||||||
|
"""Grid-based spatial index over arcgis postcodes for nearest-lookup."""
|
||||||
|
|
||||||
|
def __init__(self, lats: list[float], lngs: list[float], postcodes: list[str]):
|
||||||
|
self.grid: dict[tuple[int, int], list[tuple[float, float, str]]] = defaultdict(list)
|
||||||
|
for lat, lng, pcd in zip(lats, lngs, postcodes):
|
||||||
|
gx = int(math.floor(lng / GRID_CELL_SIZE))
|
||||||
|
gy = int(math.floor(lat / GRID_CELL_SIZE))
|
||||||
|
self.grid[(gx, gy)].append((lat, lng, pcd))
|
||||||
|
log.info("Postcode spatial index: %d cells, %d postcodes", len(self.grid), len(lats))
|
||||||
|
|
||||||
|
def nearest(self, lat: float, lng: float) -> str | None:
|
||||||
|
gx = int(math.floor(lng / GRID_CELL_SIZE))
|
||||||
|
gy = int(math.floor(lat / GRID_CELL_SIZE))
|
||||||
|
best_dist = float("inf")
|
||||||
|
best_pcd = None
|
||||||
|
for dx in range(-1, 2):
|
||||||
|
for dy in range(-1, 2):
|
||||||
|
for plat, plng, pcd in self.grid.get((gx + dx, gy + dy), []):
|
||||||
|
d = (plat - lat) ** 2 + (plng - lng) ** 2
|
||||||
|
if d < best_dist:
|
||||||
|
best_dist = d
|
||||||
|
best_pcd = pcd
|
||||||
|
return best_pcd
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Scrape status
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class ScrapeStatus:
|
||||||
|
state: str = "idle" # idle | running | done | error
|
||||||
|
channel: str = ""
|
||||||
|
outcode: str = ""
|
||||||
|
outcodes_done: int = 0
|
||||||
|
outcodes_total: int = 0
|
||||||
|
properties_buy: int = 0
|
||||||
|
properties_rent: int = 0
|
||||||
|
errors: list[str] = field(default_factory=list)
|
||||||
|
started_at: float = 0.0
|
||||||
|
finished_at: float = 0.0
|
||||||
|
|
||||||
|
|
||||||
|
status = ScrapeStatus()
|
||||||
|
status_lock = threading.Lock()
|
||||||
|
debug_data: dict = {"last_response": None, "outcode_cache": {}}
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# HTTP helpers
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
USER_AGENT = (
|
||||||
|
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
|
||||||
|
"AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Gluetun control API — runs on port 8000 inside the gluetun container.
|
||||||
|
# Since finder uses network_mode: service:gluetun, localhost IS gluetun.
|
||||||
|
GLUETUN_API = "http://127.0.0.1:8000"
|
||||||
|
_ip_rotate_lock = threading.Lock()
|
||||||
|
|
||||||
|
|
||||||
|
def rotate_ip() -> bool:
|
||||||
|
"""Ask gluetun to reconnect to a different VPN server, getting a new IP.
|
||||||
|
Returns True if the IP changed successfully."""
|
||||||
|
with _ip_rotate_lock:
|
||||||
|
log.info("Rotating VPN IP via gluetun...")
|
||||||
|
try:
|
||||||
|
# Get current IP
|
||||||
|
with httpx.Client(timeout=10) as ctl:
|
||||||
|
old_ip_resp = ctl.get(f"{GLUETUN_API}/v1/publicip/ip")
|
||||||
|
old_ip = old_ip_resp.json().get("public_ip", "unknown") if old_ip_resp.status_code == 200 else "unknown"
|
||||||
|
log.info("Current IP: %s", old_ip)
|
||||||
|
|
||||||
|
# Trigger server change — PUT with empty JSON body picks a random server
|
||||||
|
resp = ctl.put(f"{GLUETUN_API}/v1/vpn/status", json={"status": "stopped"})
|
||||||
|
if resp.status_code != 200:
|
||||||
|
log.error("Failed to stop VPN: %d %s", resp.status_code, resp.text)
|
||||||
|
return False
|
||||||
|
time.sleep(2)
|
||||||
|
|
||||||
|
resp = ctl.put(f"{GLUETUN_API}/v1/vpn/status", json={"status": "running"})
|
||||||
|
if resp.status_code != 200:
|
||||||
|
log.error("Failed to start VPN: %d %s", resp.status_code, resp.text)
|
||||||
|
return False
|
||||||
|
|
||||||
|
# Wait for reconnection
|
||||||
|
for _ in range(30):
|
||||||
|
time.sleep(2)
|
||||||
|
try:
|
||||||
|
with httpx.Client(timeout=10) as ctl:
|
||||||
|
new_ip_resp = ctl.get(f"{GLUETUN_API}/v1/publicip/ip")
|
||||||
|
if new_ip_resp.status_code == 200:
|
||||||
|
new_ip = new_ip_resp.json().get("public_ip", "")
|
||||||
|
if new_ip and new_ip != old_ip:
|
||||||
|
log.info("IP rotated: %s → %s", old_ip, new_ip)
|
||||||
|
return True
|
||||||
|
except Exception:
|
||||||
|
pass # VPN still reconnecting
|
||||||
|
|
||||||
|
log.warning("IP rotation timed out (may still be same IP)")
|
||||||
|
return False
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
log.error("IP rotation failed: %s", e)
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def make_client() -> httpx.Client:
|
||||||
|
return httpx.Client(
|
||||||
|
timeout=30,
|
||||||
|
headers={"User-Agent": USER_AGENT, "Accept": "application/json"},
|
||||||
|
follow_redirects=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_with_retry(
|
||||||
|
client: httpx.Client, url: str, params: dict | None = None, on_403: bool = True
|
||||||
|
) -> dict | None:
|
||||||
|
"""GET JSON with retries on 429/5xx/connection errors. Returns None on permanent failure.
|
||||||
|
On 403, triggers IP rotation and retries once."""
|
||||||
|
for attempt in range(MAX_RETRIES):
|
||||||
|
try:
|
||||||
|
resp = client.get(url, params=params)
|
||||||
|
if resp.status_code == 200:
|
||||||
|
return resp.json()
|
||||||
|
if resp.status_code == 403 and on_403:
|
||||||
|
log.warning("HTTP 403 — IP likely blocked, rotating...")
|
||||||
|
if rotate_ip():
|
||||||
|
# Retry once with new IP (but don't recurse on 403 again)
|
||||||
|
return fetch_with_retry(client, url, params, on_403=False)
|
||||||
|
log.error("IP rotation failed, giving up on %s", url)
|
||||||
|
return None
|
||||||
|
if resp.status_code in (429, 500, 502, 503, 504):
|
||||||
|
delay = RETRY_BASE_DELAY * (2**attempt) + random.uniform(0, 1)
|
||||||
|
log.warning("HTTP %d from %s, retry %d/%d in %.1fs", resp.status_code, url, attempt + 1, MAX_RETRIES, delay)
|
||||||
|
time.sleep(delay)
|
||||||
|
continue
|
||||||
|
log.error("HTTP %d from %s (non-retryable)", resp.status_code, url)
|
||||||
|
return None
|
||||||
|
except (httpx.ConnectError, httpx.ReadTimeout, httpx.WriteTimeout, httpx.PoolTimeout) as e:
|
||||||
|
delay = RETRY_BASE_DELAY * (2**attempt) + random.uniform(0, 1)
|
||||||
|
log.warning("%s from %s, retry %d/%d in %.1fs", type(e).__name__, url, attempt + 1, MAX_RETRIES, delay)
|
||||||
|
time.sleep(delay)
|
||||||
|
log.error("All %d retries exhausted for %s", MAX_RETRIES, url)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Rightmove API
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def resolve_outcode_id(client: httpx.Client, outcode: str) -> str | None:
|
||||||
|
"""Look up Rightmove's internal ID for an outcode via typeahead API."""
|
||||||
|
if outcode in debug_data["outcode_cache"]:
|
||||||
|
return debug_data["outcode_cache"][outcode]
|
||||||
|
|
||||||
|
data = fetch_with_retry(client, TYPEAHEAD_URL, {"query": outcode, "limit": "10", "exclude": "STREET"})
|
||||||
|
if not data:
|
||||||
|
return None
|
||||||
|
|
||||||
|
for match in data.get("matches", []):
|
||||||
|
if match.get("type") == "OUTCODE" and match.get("displayName") == outcode:
|
||||||
|
rid = str(match["id"])
|
||||||
|
debug_data["outcode_cache"][outcode] = rid
|
||||||
|
return rid
|
||||||
|
|
||||||
|
log.debug("Outcode %s not found in typeahead results", outcode)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def search_outcode(
|
||||||
|
client: httpx.Client,
|
||||||
|
outcode_id: str,
|
||||||
|
outcode: str,
|
||||||
|
channel_cfg: dict,
|
||||||
|
pc_index: PostcodeSpatialIndex,
|
||||||
|
) -> list[dict]:
|
||||||
|
"""Paginate through search results for one outcode+channel. Returns transformed properties."""
|
||||||
|
properties = []
|
||||||
|
index = 0
|
||||||
|
|
||||||
|
for page in range(MAX_PAGES_PER_OUTCODE):
|
||||||
|
params = {
|
||||||
|
"useLocationIdentifier": "true",
|
||||||
|
"locationIdentifier": f"OUTCODE^{outcode_id}",
|
||||||
|
"index": str(index),
|
||||||
|
"sortType": channel_cfg["sortType"],
|
||||||
|
"channel": channel_cfg["channel"],
|
||||||
|
"transactionType": channel_cfg["transactionType"],
|
||||||
|
}
|
||||||
|
|
||||||
|
data = fetch_with_retry(client, SEARCH_URL, params)
|
||||||
|
if not data:
|
||||||
|
log.warning("Failed to fetch page %d for %s/%s", page, outcode, channel_cfg["channel"])
|
||||||
|
break
|
||||||
|
|
||||||
|
debug_data["last_response"] = data
|
||||||
|
|
||||||
|
raw_props = data.get("properties", [])
|
||||||
|
if not raw_props:
|
||||||
|
break
|
||||||
|
|
||||||
|
for prop in raw_props:
|
||||||
|
transformed = transform_property(prop, outcode, pc_index)
|
||||||
|
if transformed:
|
||||||
|
properties.append(transformed)
|
||||||
|
|
||||||
|
# Check if there are more pages
|
||||||
|
result_count_str = data.get("resultCount", "0")
|
||||||
|
result_count = int(result_count_str.replace(",", ""))
|
||||||
|
index += PAGE_SIZE
|
||||||
|
|
||||||
|
if index >= result_count:
|
||||||
|
break
|
||||||
|
|
||||||
|
if page < MAX_PAGES_PER_OUTCODE - 1:
|
||||||
|
time.sleep(DELAY_BETWEEN_PAGES)
|
||||||
|
|
||||||
|
return properties
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Property transformation
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def parse_display_size(display_size: str | None) -> float | None:
|
||||||
|
"""Parse displaySize like '499 sq. ft.' or '4,124 sq. ft.' to sqm."""
|
||||||
|
if not display_size:
|
||||||
|
return None
|
||||||
|
# Try sq. ft. first
|
||||||
|
m = re.search(r"([\d,]+(?:\.\d+)?)\s*sq\.?\s*ft", display_size, re.IGNORECASE)
|
||||||
|
if m:
|
||||||
|
sqft = float(m.group(1).replace(",", ""))
|
||||||
|
return round(sqft * 0.092903, 1)
|
||||||
|
# Try sq. m.
|
||||||
|
m = re.search(r"([\d,]+(?:\.\d+)?)\s*sq\.?\s*m", display_size, re.IGNORECASE)
|
||||||
|
if m:
|
||||||
|
return round(float(m.group(1).replace(",", "")), 1)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def map_property_type(sub_type: str | None) -> str:
|
||||||
|
"""Map propertySubType to canonical type."""
|
||||||
|
if not sub_type:
|
||||||
|
return "Other"
|
||||||
|
canonical = PROPERTY_TYPE_MAP.get(sub_type)
|
||||||
|
if canonical:
|
||||||
|
return canonical
|
||||||
|
log.warning("Unknown propertySubType: %r — mapping to Other", sub_type)
|
||||||
|
return "Other"
|
||||||
|
|
||||||
|
|
||||||
|
def extract_tenure(tenure_obj: dict | None) -> str | None:
|
||||||
|
"""Extract tenure string from tenure object."""
|
||||||
|
if not tenure_obj:
|
||||||
|
return None
|
||||||
|
tt = tenure_obj.get("tenureType", "")
|
||||||
|
if tt == "FREEHOLD":
|
||||||
|
return "Freehold"
|
||||||
|
if tt == "LEASEHOLD":
|
||||||
|
return "Leasehold"
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def fix_coords(lat: float, lng: float) -> tuple[float, float]:
|
||||||
|
"""Swap lat/lng if they look reversed. England: lat ~49–56, lng ~-7–2."""
|
||||||
|
if 49 <= lat <= 56 and -7 <= lng <= 2:
|
||||||
|
return lat, lng
|
||||||
|
if 49 <= lng <= 56 and -7 <= lat <= 2:
|
||||||
|
log.debug("Swapping reversed coords: lat=%.4f lng=%.4f → lat=%.4f lng=%.4f", lat, lng, lng, lat)
|
||||||
|
return lng, lat
|
||||||
|
log.warning("Coords outside England bounds even after swap attempt: lat=%.4f lng=%.4f", lat, lng)
|
||||||
|
return lat, lng
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_price(amount: int, frequency: str) -> int:
|
||||||
|
"""Normalize price to monthly for rentals (weekly × 52/12, yearly ÷ 12)."""
|
||||||
|
if frequency == "weekly":
|
||||||
|
return round(amount * 52 / 12)
|
||||||
|
if frequency == "yearly":
|
||||||
|
return round(amount / 12)
|
||||||
|
return amount
|
||||||
|
|
||||||
|
|
||||||
|
def transform_property(prop: dict, outcode: str, pc_index: PostcodeSpatialIndex) -> dict | None:
|
||||||
|
"""Transform a raw Rightmove property dict into our output schema."""
|
||||||
|
loc = prop.get("location")
|
||||||
|
if not loc:
|
||||||
|
return None
|
||||||
|
raw_lat = loc.get("latitude")
|
||||||
|
raw_lng = loc.get("longitude")
|
||||||
|
if raw_lat is None or raw_lng is None:
|
||||||
|
return None
|
||||||
|
|
||||||
|
lat, lng = fix_coords(raw_lat, raw_lng)
|
||||||
|
|
||||||
|
price_obj = prop.get("price", {})
|
||||||
|
amount = price_obj.get("amount")
|
||||||
|
if amount is None:
|
||||||
|
return None
|
||||||
|
frequency = price_obj.get("frequency", "")
|
||||||
|
price = normalize_price(int(amount), frequency)
|
||||||
|
|
||||||
|
display_prices = price_obj.get("displayPrices", [])
|
||||||
|
price_qualifier = display_prices[0].get("displayPriceQualifier", "") if display_prices else ""
|
||||||
|
|
||||||
|
sub_type = prop.get("propertySubType", "")
|
||||||
|
bedrooms = prop.get("bedrooms", 0) or 0
|
||||||
|
bathrooms = prop.get("bathrooms", 0) or 0
|
||||||
|
|
||||||
|
key_features = [kf.get("description", "") for kf in prop.get("keyFeatures", []) if kf.get("description")]
|
||||||
|
|
||||||
|
listing_update = prop.get("listingUpdate", {})
|
||||||
|
update_date = listing_update.get("listingUpdateDate", "")
|
||||||
|
|
||||||
|
postcode = pc_index.nearest(lat, lng)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"id": prop.get("id"),
|
||||||
|
"bedrooms": bedrooms,
|
||||||
|
"bathrooms": bathrooms,
|
||||||
|
"total_rooms": bedrooms + bathrooms,
|
||||||
|
"longitude": lng,
|
||||||
|
"latitude": lat,
|
||||||
|
"postcode": postcode,
|
||||||
|
"address": prop.get("displayAddress", ""),
|
||||||
|
"tenure": extract_tenure(prop.get("tenure")),
|
||||||
|
"property_type": map_property_type(sub_type),
|
||||||
|
"property_sub_type": sub_type or "Unknown",
|
||||||
|
"price": price,
|
||||||
|
"price_frequency": frequency,
|
||||||
|
"price_qualifier": price_qualifier,
|
||||||
|
"floorspace_sqm": parse_display_size(prop.get("displaySize")),
|
||||||
|
"url": RIGHTMOVE_BASE + prop.get("propertyUrl", ""),
|
||||||
|
"features": key_features,
|
||||||
|
"first_visible_date": prop.get("firstVisibleDate", ""),
|
||||||
|
"update_date": update_date,
|
||||||
|
"outcode": outcode,
|
||||||
|
"house_share": sub_type == "House Share",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Parquet writing
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def write_parquet(properties: list[dict], path: Path) -> None:
|
||||||
|
"""Write properties list to parquet using Polars."""
|
||||||
|
if not properties:
|
||||||
|
log.warning("No properties to write to %s", path)
|
||||||
|
return
|
||||||
|
|
||||||
|
df = pl.DataFrame(
|
||||||
|
{
|
||||||
|
"id": [p["id"] for p in properties],
|
||||||
|
"bedrooms": [p["bedrooms"] for p in properties],
|
||||||
|
"bathrooms": [p["bathrooms"] for p in properties],
|
||||||
|
"total_rooms": [p["total_rooms"] for p in properties],
|
||||||
|
"longitude": [p["longitude"] for p in properties],
|
||||||
|
"latitude": [p["latitude"] for p in properties],
|
||||||
|
"postcode": [p["postcode"] for p in properties],
|
||||||
|
"address": [p["address"] for p in properties],
|
||||||
|
"tenure": [p["tenure"] for p in properties],
|
||||||
|
"property_type": [p["property_type"] for p in properties],
|
||||||
|
"property_sub_type": [p["property_sub_type"] for p in properties],
|
||||||
|
"price": [p["price"] for p in properties],
|
||||||
|
"price_frequency": [p["price_frequency"] for p in properties],
|
||||||
|
"price_qualifier": [p["price_qualifier"] for p in properties],
|
||||||
|
"floorspace_sqm": [p["floorspace_sqm"] for p in properties],
|
||||||
|
"url": [p["url"] for p in properties],
|
||||||
|
"features": [p["features"] for p in properties],
|
||||||
|
"first_visible_date": [p["first_visible_date"] for p in properties],
|
||||||
|
"update_date": [p["update_date"] for p in properties],
|
||||||
|
"outcode": [p["outcode"] for p in properties],
|
||||||
|
"house_share": [p["house_share"] for p in properties],
|
||||||
|
},
|
||||||
|
schema={
|
||||||
|
"id": pl.Int64,
|
||||||
|
"bedrooms": pl.Int32,
|
||||||
|
"bathrooms": pl.Int32,
|
||||||
|
"total_rooms": pl.Int32,
|
||||||
|
"longitude": pl.Float64,
|
||||||
|
"latitude": pl.Float64,
|
||||||
|
"postcode": pl.Utf8,
|
||||||
|
"address": pl.Utf8,
|
||||||
|
"tenure": pl.Utf8,
|
||||||
|
"property_type": pl.Utf8,
|
||||||
|
"property_sub_type": pl.Utf8,
|
||||||
|
"price": pl.Int64,
|
||||||
|
"price_frequency": pl.Utf8,
|
||||||
|
"price_qualifier": pl.Utf8,
|
||||||
|
"floorspace_sqm": pl.Float64,
|
||||||
|
"url": pl.Utf8,
|
||||||
|
"features": pl.List(pl.Utf8),
|
||||||
|
"first_visible_date": pl.Utf8,
|
||||||
|
"update_date": pl.Utf8,
|
||||||
|
"outcode": pl.Utf8,
|
||||||
|
"house_share": pl.Boolean,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
df.write_parquet(path)
|
||||||
|
log.info("Wrote %d properties to %s", len(df), path)
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Scrape orchestration
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
|
||||||
|
def load_outcodes() -> list[str]:
|
||||||
|
"""Load England-only outcodes from arcgis parquet."""
|
||||||
|
log.info("Loading outcodes from %s", ARCGIS_PATH)
|
||||||
|
df = pl.read_parquet(ARCGIS_PATH, columns=["pcd", "ctry", "lat", "long"])
|
||||||
|
england = df.filter(pl.col("ctry") == "E92000001")
|
||||||
|
log.info("England postcodes: %d", len(england))
|
||||||
|
|
||||||
|
outcodes = (
|
||||||
|
england.select(pl.col("pcd").str.extract(r"^([A-Z]{1,2}\d[A-Z0-9]?)", 1).alias("outcode"))
|
||||||
|
.drop_nulls()
|
||||||
|
.get_column("outcode")
|
||||||
|
.unique()
|
||||||
|
.sort()
|
||||||
|
.to_list()
|
||||||
|
)
|
||||||
|
log.info("Unique England outcodes: %d", len(outcodes))
|
||||||
|
return outcodes
|
||||||
|
|
||||||
|
|
||||||
|
def build_postcode_index() -> PostcodeSpatialIndex:
|
||||||
|
"""Build spatial index from arcgis England postcodes."""
|
||||||
|
log.info("Building postcode spatial index from %s", ARCGIS_PATH)
|
||||||
|
df = pl.read_parquet(ARCGIS_PATH, columns=["pcd", "ctry", "lat", "long"])
|
||||||
|
england = df.filter(pl.col("ctry") == "E92000001").drop_nulls(subset=["lat", "long"])
|
||||||
|
return PostcodeSpatialIndex(
|
||||||
|
england.get_column("lat").to_list(),
|
||||||
|
england.get_column("long").to_list(),
|
||||||
|
england.get_column("pcd").to_list(),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def run_scrape(outcodes: list[str], pc_index: PostcodeSpatialIndex) -> None:
|
||||||
|
"""Main scrape loop — runs in background thread."""
|
||||||
|
global status
|
||||||
|
with status_lock:
|
||||||
|
status.state = "running"
|
||||||
|
status.started_at = time.time()
|
||||||
|
status.errors = []
|
||||||
|
status.properties_buy = 0
|
||||||
|
status.properties_rent = 0
|
||||||
|
|
||||||
|
# Shuffle for geographic diversity
|
||||||
|
shuffled = list(outcodes)
|
||||||
|
random.seed(SEED)
|
||||||
|
random.shuffle(shuffled)
|
||||||
|
|
||||||
|
client = make_client()
|
||||||
|
|
||||||
|
try:
|
||||||
|
for channel_cfg in CHANNELS:
|
||||||
|
channel_name = channel_cfg["channel"]
|
||||||
|
file_suffix = "buy" if channel_name == "BUY" else "rent"
|
||||||
|
all_properties: dict[int, dict] = {} # dedup by id
|
||||||
|
|
||||||
|
with status_lock:
|
||||||
|
status.channel = channel_name
|
||||||
|
status.outcodes_done = 0
|
||||||
|
status.outcodes_total = len(shuffled)
|
||||||
|
|
||||||
|
log.info("=== Starting %s channel (%d outcodes) ===", channel_name, len(shuffled))
|
||||||
|
|
||||||
|
for i, outcode in enumerate(shuffled):
|
||||||
|
with status_lock:
|
||||||
|
status.outcode = outcode
|
||||||
|
status.outcodes_done = i
|
||||||
|
|
||||||
|
log.debug("Outcode %s (%d/%d) — %d properties so far",
|
||||||
|
outcode, i + 1, len(shuffled), len(all_properties))
|
||||||
|
|
||||||
|
try:
|
||||||
|
outcode_id = resolve_outcode_id(client, outcode)
|
||||||
|
if not outcode_id:
|
||||||
|
log.debug("No Rightmove ID for outcode %s, skipping", outcode)
|
||||||
|
continue
|
||||||
|
|
||||||
|
props = search_outcode(client, outcode_id, outcode, channel_cfg, pc_index)
|
||||||
|
for p in props:
|
||||||
|
pid = p["id"]
|
||||||
|
if pid not in all_properties:
|
||||||
|
all_properties[pid] = p
|
||||||
|
|
||||||
|
with status_lock:
|
||||||
|
if channel_name == "BUY":
|
||||||
|
status.properties_buy = len(all_properties)
|
||||||
|
else:
|
||||||
|
status.properties_rent = len(all_properties)
|
||||||
|
|
||||||
|
log.info("Outcode %s: got %d properties (total: %d)", outcode, len(props), len(all_properties))
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
msg = f"Error scraping {outcode}/{channel_name}: {e}"
|
||||||
|
log.error(msg)
|
||||||
|
with status_lock:
|
||||||
|
status.errors.append(msg)
|
||||||
|
|
||||||
|
if i < len(shuffled) - 1:
|
||||||
|
time.sleep(DELAY_BETWEEN_OUTCODES)
|
||||||
|
|
||||||
|
# Write parquet
|
||||||
|
deduped = list(all_properties.values())
|
||||||
|
output_path = DATA_DIR / f"rightmove_{file_suffix}.parquet"
|
||||||
|
write_parquet(deduped, output_path)
|
||||||
|
|
||||||
|
with status_lock:
|
||||||
|
if channel_name == "BUY":
|
||||||
|
status.properties_buy = len(deduped)
|
||||||
|
else:
|
||||||
|
status.properties_rent = len(deduped)
|
||||||
|
status.outcodes_done = len(shuffled)
|
||||||
|
|
||||||
|
log.info("=== %s channel complete: %d unique properties ===", channel_name, len(deduped))
|
||||||
|
|
||||||
|
with status_lock:
|
||||||
|
status.state = "done"
|
||||||
|
status.finished_at = time.time()
|
||||||
|
elapsed = status.finished_at - status.started_at
|
||||||
|
log.info("Scrape complete in %.0fs — buy: %d, rent: %d",
|
||||||
|
elapsed, status.properties_buy, status.properties_rent)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
log.exception("Fatal scrape error")
|
||||||
|
with status_lock:
|
||||||
|
status.state = "error"
|
||||||
|
status.errors.append(f"Fatal: {e}")
|
||||||
|
status.finished_at = time.time()
|
||||||
|
finally:
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Startup: load data
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
log.info("Loading arcgis data...")
|
||||||
|
OUTCODES = load_outcodes()
|
||||||
|
PC_INDEX = build_postcode_index()
|
||||||
|
log.info("Ready — %d outcodes, postcode index built", len(OUTCODES))
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Flask app
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
app = Flask(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
@app.route("/run", methods=["POST"])
|
||||||
|
def trigger_run():
|
||||||
|
with status_lock:
|
||||||
|
if status.state == "running":
|
||||||
|
return jsonify({"error": "Scrape already running"}), 409
|
||||||
|
status.state = "running"
|
||||||
|
|
||||||
|
thread = threading.Thread(target=run_scrape, args=(OUTCODES, PC_INDEX), daemon=True)
|
||||||
|
thread.start()
|
||||||
|
return jsonify({"message": "Scrape started"}), 200
|
||||||
|
|
||||||
|
|
||||||
|
@app.route("/status")
|
||||||
|
def get_status():
|
||||||
|
with status_lock:
|
||||||
|
elapsed = 0.0
|
||||||
|
if status.started_at:
|
||||||
|
end = status.finished_at if status.finished_at else time.time()
|
||||||
|
elapsed = end - status.started_at
|
||||||
|
return jsonify({
|
||||||
|
"state": status.state,
|
||||||
|
"channel": status.channel,
|
||||||
|
"outcode": status.outcode,
|
||||||
|
"outcodes_done": status.outcodes_done,
|
||||||
|
"outcodes_total": status.outcodes_total,
|
||||||
|
"properties_buy": status.properties_buy,
|
||||||
|
"properties_rent": status.properties_rent,
|
||||||
|
"errors": status.errors[-20:], # last 20 errors
|
||||||
|
"elapsed_seconds": round(elapsed, 1),
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
@app.route("/debug")
|
||||||
|
def get_debug():
|
||||||
|
return jsonify({
|
||||||
|
"last_response": debug_data["last_response"],
|
||||||
|
"outcode_cache_size": len(debug_data["outcode_cache"]),
|
||||||
|
"outcode_cache_sample": dict(list(debug_data["outcode_cache"].items())[:20]),
|
||||||
|
})
|
||||||
|
|
||||||
|
|
||||||
|
@app.route("/data/<filename>")
|
||||||
|
def serve_data(filename):
|
||||||
|
if not filename.endswith(".parquet"):
|
||||||
|
return jsonify({"error": "Only parquet files served"}), 400
|
||||||
|
return send_from_directory(DATA_DIR, filename)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
app.run(host="0.0.0.0", port=1234, debug=False)
|
||||||
6
finder/onthemarket/explain.md
Normal file
6
finder/onthemarket/explain.md
Normal file
|
|
@ -0,0 +1,6 @@
|
||||||
|
|
||||||
|
Hit the following url with the outcode as the location-id and the page. So for E13, page 2 it's:
|
||||||
|
|
||||||
|
https://www.onthemarket.com/async/search/properties-v2/?search-type=for-sale&location-id=e13&page=2&view=map-list
|
||||||
|
|
||||||
|
and the response is in [[response.json]]
|
||||||
4256
finder/onthemarket/response.json
Normal file
4256
finder/onthemarket/response.json
Normal file
File diff suppressed because it is too large
Load diff
9
finder/pyproject.toml
Normal file
9
finder/pyproject.toml
Normal file
|
|
@ -0,0 +1,9 @@
|
||||||
|
[project]
|
||||||
|
name = "finder"
|
||||||
|
version = "0.1.0"
|
||||||
|
requires-python = ">=3.12"
|
||||||
|
dependencies = [
|
||||||
|
"flask",
|
||||||
|
"httpx",
|
||||||
|
"polars",
|
||||||
|
]
|
||||||
10918
finder/rightmove/buy.json
Normal file
10918
finder/rightmove/buy.json
Normal file
File diff suppressed because it is too large
Load diff
52
finder/rightmove/explain.md
Normal file
52
finder/rightmove/explain.md
Normal file
|
|
@ -0,0 +1,52 @@
|
||||||
|
The API works as follows, you must search for outcodes, such as E11, then hit https://los.rightmove.co.uk/typeahead?query=E11&limit=10&exclude=STREET which will return something like:
|
||||||
|
|
||||||
|
{
|
||||||
|
"matches": [
|
||||||
|
{
|
||||||
|
"id": "746",
|
||||||
|
"type": "OUTCODE",
|
||||||
|
"displayName": "E11",
|
||||||
|
"highlighting": "<span class='highlightLetter'>E11</span>",
|
||||||
|
"highlights": [
|
||||||
|
{
|
||||||
|
"text": "E11",
|
||||||
|
"highlighted": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "749",
|
||||||
|
"type": "OUTCODE",
|
||||||
|
"displayName": "E14",
|
||||||
|
"highlighting": "displayName",
|
||||||
|
"highlights": []
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "752",
|
||||||
|
"type": "OUTCODE",
|
||||||
|
"displayName": "E17",
|
||||||
|
"highlighting": "displayName",
|
||||||
|
"highlights": []
|
||||||
|
},
|
||||||
|
...
|
||||||
|
]
|
||||||
|
}
|
||||||
|
|
||||||
|
We need to find the id of the object which has "type": "OUTCODE", and displayName matching the outcode we searched for, in this case E11, which is 746. Then we can hit the search endpoint with that id, and it will return the properties for that outcode:
|
||||||
|
|
||||||
|
https://www.rightmove.co.uk/api/property-search/listing/search?useLocationIdentifier=true&locationIdentifier=OUTCODE%5E746&buy=For+sale&_includeSSTC=on&index=0&sortType=2&channel=BUY&transactionType=BUY&displayLocationIdentifier=E12.html
|
||||||
|
|
||||||
|
You can see the example response to this at [[buy.json]]
|
||||||
|
|
||||||
|
You must set locationIdentifier=OUTCODE%5E{id} where id is 746 in this case, so it's 746 locationIdentifier=OUTCODE%5E746. Paging works by increasing index by the number of results per page, which is 24. So the next page would be index=24, then index=48, etc.
|
||||||
|
|
||||||
|
|
||||||
|
The rental endpoint works similarly:
|
||||||
|
|
||||||
|
https://www.rightmove.co.uk/api/property-search/listing/search?locationIdentifier=OUTCODE%5E745&index=0&sortType=6&channel=RENT&transactionType=LETTING&displayLocationIdentifier=E16.html
|
||||||
|
|
||||||
|
https://www.rightmove.co.uk/api/property-search/listing/search?locationIdentifier=OUTCODE%5E752&index=48&sortType=6&channel=RENT&transactionType=LETTING&displayLocationIdentifier=E17.html
|
||||||
|
|
||||||
|
|
||||||
|
See a response example for the rental endpoint at [[rent.json]]
|
||||||
|
|
||||||
8247
finder/rightmove/rental.json
Normal file
8247
finder/rightmove/rental.json
Normal file
File diff suppressed because it is too large
Load diff
|
|
@ -1,7 +1,7 @@
|
||||||
import { useState, useEffect, useCallback, useRef, useMemo } from 'react';
|
import { useState, useEffect, useCallback, useRef, useMemo } from 'react';
|
||||||
import MapComponent from '../map/Map';
|
import MapComponent from '../map/Map';
|
||||||
import { Slider } from '../ui/Slider';
|
import { Slider } from '../ui/Slider';
|
||||||
import { apiUrl, authHeaders, logNonAbortError } from '../../lib/api';
|
import { apiUrl, assertOk, authHeaders, logNonAbortError } from '../../lib/api';
|
||||||
import { formatValue } from '../../lib/format';
|
import { formatValue } from '../../lib/format';
|
||||||
import { FEATURE_GRADIENT, DENSITY_GRADIENT, DENSITY_GRADIENT_DARK } from '../../lib/consts';
|
import { FEATURE_GRADIENT, DENSITY_GRADIENT, DENSITY_GRADIENT_DARK } from '../../lib/consts';
|
||||||
import { gradientToCss } from '../../lib/utils';
|
import { gradientToCss } from '../../lib/utils';
|
||||||
|
|
@ -88,7 +88,10 @@ export default function HomeDemo({ features, theme }: HomeDemoProps) {
|
||||||
abortRef.current = new AbortController();
|
abortRef.current = new AbortController();
|
||||||
setFetching(true);
|
setFetching(true);
|
||||||
fetch(apiUrl('hexagons', params), authHeaders({ signal: abortRef.current.signal }))
|
fetch(apiUrl('hexagons', params), authHeaders({ signal: abortRef.current.signal }))
|
||||||
.then((res) => res.json())
|
.then((res) => {
|
||||||
|
assertOk(res, 'hexagons');
|
||||||
|
return res.json();
|
||||||
|
})
|
||||||
.then((data: { features: HexagonData[] }) => {
|
.then((data: { features: HexagonData[] }) => {
|
||||||
setHexData(data.features);
|
setHexData(data.features);
|
||||||
setLoading(false);
|
setLoading(false);
|
||||||
|
|
@ -142,7 +145,10 @@ export default function HomeDemo({ features, theme }: HomeDemoProps) {
|
||||||
dragAbortRef.current?.abort();
|
dragAbortRef.current?.abort();
|
||||||
dragAbortRef.current = new AbortController();
|
dragAbortRef.current = new AbortController();
|
||||||
fetch(apiUrl('hexagons', params), authHeaders({ signal: dragAbortRef.current.signal }))
|
fetch(apiUrl('hexagons', params), authHeaders({ signal: dragAbortRef.current.signal }))
|
||||||
.then((res) => res.json())
|
.then((res) => {
|
||||||
|
assertOk(res, 'hexagons');
|
||||||
|
return res.json();
|
||||||
|
})
|
||||||
.then((data: { features: HexagonData[] }) => setDragHexData(data.features))
|
.then((data: { features: HexagonData[] }) => setDragHexData(data.features))
|
||||||
.catch((err) => logNonAbortError('Failed to fetch demo drag data', err));
|
.catch((err) => logNonAbortError('Failed to fetch demo drag data', err));
|
||||||
},
|
},
|
||||||
|
|
|
||||||
|
|
@ -4,10 +4,11 @@ import { SpinnerIcon } from '../ui/icons/SpinnerIcon';
|
||||||
interface AiFilterInputProps {
|
interface AiFilterInputProps {
|
||||||
loading: boolean;
|
loading: boolean;
|
||||||
error: string | null;
|
error: string | null;
|
||||||
|
notes: string | null;
|
||||||
onSubmit: (query: string) => void;
|
onSubmit: (query: string) => void;
|
||||||
}
|
}
|
||||||
|
|
||||||
export default memo(function AiFilterInput({ loading, error, onSubmit }: AiFilterInputProps) {
|
export default memo(function AiFilterInput({ loading, error, notes, onSubmit }: AiFilterInputProps) {
|
||||||
const [query, setQuery] = useState('');
|
const [query, setQuery] = useState('');
|
||||||
|
|
||||||
const handleSubmit = useCallback(
|
const handleSubmit = useCallback(
|
||||||
|
|
@ -48,6 +49,11 @@ export default memo(function AiFilterInput({ loading, error, onSubmit }: AiFilte
|
||||||
{error}
|
{error}
|
||||||
</p>
|
</p>
|
||||||
)}
|
)}
|
||||||
|
{notes && !error && (
|
||||||
|
<p className="mt-1 text-xs text-warm-500 dark:text-warm-400 italic">
|
||||||
|
{notes}
|
||||||
|
</p>
|
||||||
|
)}
|
||||||
</div>
|
</div>
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
|
||||||
|
|
@ -155,10 +155,10 @@ export default function AreaPane({
|
||||||
const stackedEnumCharts = STACKED_ENUM_GROUPS[group.name];
|
const stackedEnumCharts = STACKED_ENUM_GROUPS[group.name];
|
||||||
|
|
||||||
// Features that are part of a stacked enum config (rendered as compact charts)
|
// Features that are part of a stacked enum config (rendered as compact charts)
|
||||||
const stackedEnumFeatureNames = new Set(
|
const stackedEnumFeatureNames = new Set<string>(
|
||||||
(stackedEnumCharts?.flatMap((c) =>
|
stackedEnumCharts?.flatMap((c) =>
|
||||||
[c.feature, ...c.components].filter(Boolean)
|
[c.feature, ...c.components].filter((s): s is string => Boolean(s))
|
||||||
) as string[]) ?? []
|
) ?? []
|
||||||
);
|
);
|
||||||
|
|
||||||
const isExpanded = !collapsedGroups.has(group.name);
|
const isExpanded = !collapsedGroups.has(group.name);
|
||||||
|
|
|
||||||
|
|
@ -11,6 +11,7 @@ import { FeatureActions } from '../ui/FeatureIcons';
|
||||||
import { FeatureLabel } from '../ui/FeatureLabel';
|
import { FeatureLabel } from '../ui/FeatureLabel';
|
||||||
import { RouteIcon, PlusIcon } from '../ui/icons';
|
import { RouteIcon, PlusIcon } from '../ui/icons';
|
||||||
import { IconButton } from '../ui/IconButton';
|
import { IconButton } from '../ui/IconButton';
|
||||||
|
import { TRANSPORT_MODES, MODE_LABELS, type TransportMode } from '../../hooks/useTravelTime';
|
||||||
|
|
||||||
interface FeatureBrowserProps {
|
interface FeatureBrowserProps {
|
||||||
availableFeatures: FeatureMeta[];
|
availableFeatures: FeatureMeta[];
|
||||||
|
|
@ -21,8 +22,8 @@ interface FeatureBrowserProps {
|
||||||
onNavigateToSource?: (slug: string, featureName: string) => void;
|
onNavigateToSource?: (slug: string, featureName: string) => void;
|
||||||
openInfoFeature?: string | null;
|
openInfoFeature?: string | null;
|
||||||
onClearOpenInfoFeature?: () => void;
|
onClearOpenInfoFeature?: () => void;
|
||||||
travelTimeEnabled?: boolean;
|
activeTravelModes: TransportMode[];
|
||||||
onEnableTravelTime?: () => void;
|
onEnableTravelMode: (mode: TransportMode) => void;
|
||||||
}
|
}
|
||||||
|
|
||||||
export default function FeatureBrowser({
|
export default function FeatureBrowser({
|
||||||
|
|
@ -34,8 +35,8 @@ export default function FeatureBrowser({
|
||||||
onNavigateToSource,
|
onNavigateToSource,
|
||||||
openInfoFeature,
|
openInfoFeature,
|
||||||
onClearOpenInfoFeature,
|
onClearOpenInfoFeature,
|
||||||
travelTimeEnabled,
|
activeTravelModes,
|
||||||
onEnableTravelTime,
|
onEnableTravelMode,
|
||||||
}: FeatureBrowserProps) {
|
}: FeatureBrowserProps) {
|
||||||
const [search, setSearch] = useState('');
|
const [search, setSearch] = useState('');
|
||||||
const [infoFeature, setInfoFeature] = useState<FeatureMeta | null>(null);
|
const [infoFeature, setInfoFeature] = useState<FeatureMeta | null>(null);
|
||||||
|
|
@ -60,32 +61,42 @@ export default function FeatureBrowser({
|
||||||
// When searching, expand all groups so results are visible
|
// When searching, expand all groups so results are visible
|
||||||
const isSearching = search.length > 0;
|
const isSearching = search.length > 0;
|
||||||
|
|
||||||
|
// Inactive modes available to add
|
||||||
|
const inactiveModes = useMemo(
|
||||||
|
() => TRANSPORT_MODES.filter((m) => !activeTravelModes.includes(m)),
|
||||||
|
[activeTravelModes]
|
||||||
|
);
|
||||||
|
|
||||||
|
const showTravelModes =
|
||||||
|
inactiveModes.length > 0 &&
|
||||||
|
(!search || 'travel time journey commute car bicycle walking transit'.includes(search.toLowerCase()));
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<>
|
<>
|
||||||
<div className="shrink-0 p-2 border-b border-warm-200 dark:border-navy-700">
|
<div className="shrink-0 p-2 border-b border-warm-200 dark:border-navy-700">
|
||||||
<SearchInput value={search} onChange={setSearch} placeholder="Search features..." />
|
<SearchInput value={search} onChange={setSearch} placeholder="Search features..." />
|
||||||
</div>
|
</div>
|
||||||
<div className="md:min-h-0 md:flex-1 md:overflow-y-auto flex flex-col">
|
<div className="md:min-h-0 md:flex-1 md:overflow-y-auto flex flex-col">
|
||||||
{!travelTimeEnabled && onEnableTravelTime && (!search || 'travel time journey commute'.includes(search.toLowerCase())) && (
|
{showTravelModes && inactiveModes.map((mode) => (
|
||||||
<div className="shrink-0 border-b border-warm-200 dark:border-warm-700">
|
<div key={mode} className="shrink-0 border-b border-warm-200 dark:border-warm-700">
|
||||||
<div className="flex items-start justify-between px-3 py-2 hover:bg-teal-50 dark:hover:bg-teal-900/30 cursor-pointer">
|
<div className="flex items-start justify-between px-3 py-2 hover:bg-teal-50 dark:hover:bg-teal-900/30 cursor-pointer">
|
||||||
<div className="flex items-center gap-2 min-w-0" onClick={onEnableTravelTime}>
|
<div className="flex items-center gap-2 min-w-0" onClick={() => onEnableTravelMode(mode)}>
|
||||||
<RouteIcon className="w-4 h-4 text-teal-600 dark:text-teal-400 shrink-0" />
|
<RouteIcon className="w-4 h-4 text-teal-600 dark:text-teal-400 shrink-0" />
|
||||||
<div className="min-w-0">
|
<div className="min-w-0">
|
||||||
<span className="text-sm font-medium text-navy-950 dark:text-warm-100">
|
<span className="text-sm font-medium text-navy-950 dark:text-warm-100">
|
||||||
Travel Time
|
Travel Time ({MODE_LABELS[mode]})
|
||||||
</span>
|
</span>
|
||||||
<span className="text-xs text-warm-400 dark:text-warm-500 block">
|
<span className="text-xs text-warm-400 dark:text-warm-500 block">
|
||||||
Color by journey time to a destination
|
Color by journey time to a destination
|
||||||
</span>
|
</span>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
<IconButton onClick={() => onEnableTravelTime()} title="Add travel time">
|
<IconButton onClick={() => onEnableTravelMode(mode)} title={`Add ${MODE_LABELS[mode]} travel time`}>
|
||||||
<PlusIcon className="w-3.5 h-3.5" />
|
<PlusIcon className="w-3.5 h-3.5" />
|
||||||
</IconButton>
|
</IconButton>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
)}
|
))}
|
||||||
{grouped.map((group) => {
|
{grouped.map((group) => {
|
||||||
const isExpanded = isSearching || expandedGroups.has(group.name);
|
const isExpanded = isSearching || expandedGroups.has(group.name);
|
||||||
return (
|
return (
|
||||||
|
|
@ -128,7 +139,7 @@ export default function FeatureBrowser({
|
||||||
</div>
|
</div>
|
||||||
);
|
);
|
||||||
})}
|
})}
|
||||||
{grouped.length === 0 ? (
|
{grouped.length === 0 && !showTravelModes ? (
|
||||||
<EmptyState
|
<EmptyState
|
||||||
icon={<FilterIcon className="w-8 h-8 text-warm-300 dark:text-warm-600" />}
|
icon={<FilterIcon className="w-8 h-8 text-warm-300 dark:text-warm-600" />}
|
||||||
title={search ? 'No matching features' : 'All features are active'}
|
title={search ? 'No matching features' : 'All features are active'}
|
||||||
|
|
|
||||||
|
|
@ -17,18 +17,25 @@ import { FeatureLabel } from '../ui/FeatureLabel';
|
||||||
import AiFilterInput from './AiFilterInput';
|
import AiFilterInput from './AiFilterInput';
|
||||||
import FeatureBrowser from './FeatureBrowser';
|
import FeatureBrowser from './FeatureBrowser';
|
||||||
import { TravelTimeCard } from './TravelTimeCard';
|
import { TravelTimeCard } from './TravelTimeCard';
|
||||||
import type { TransportMode } from '../../hooks/useTravelTime';
|
import {
|
||||||
|
TRANSPORT_MODES,
|
||||||
|
type TransportMode,
|
||||||
|
type TravelTimeEntries,
|
||||||
|
} from '../../hooks/useTravelTime';
|
||||||
|
|
||||||
function SliderLabels({
|
function SliderLabels({
|
||||||
min,
|
min,
|
||||||
max,
|
max,
|
||||||
value,
|
value,
|
||||||
displayValues,
|
displayValues,
|
||||||
|
absoluteMax,
|
||||||
}: {
|
}: {
|
||||||
min: number;
|
min: number;
|
||||||
max: number;
|
max: number;
|
||||||
value: [number, number];
|
value: [number, number];
|
||||||
displayValues?: [number, number];
|
displayValues?: [number, number];
|
||||||
|
/** When true and slider is at max, append "+" to indicate unrestricted upper bound */
|
||||||
|
absoluteMax?: boolean;
|
||||||
}) {
|
}) {
|
||||||
const range = max - min || 1;
|
const range = max - min || 1;
|
||||||
const leftPct = ((value[0] - min) / range) * 100;
|
const leftPct = ((value[0] - min) / range) * 100;
|
||||||
|
|
@ -46,7 +53,7 @@ function SliderLabels({
|
||||||
className="absolute -translate-x-1/2"
|
className="absolute -translate-x-1/2"
|
||||||
style={{ left: `${rightPct}%` }}
|
style={{ left: `${rightPct}%` }}
|
||||||
>
|
>
|
||||||
{formatFilterValue(labels[1])}
|
{formatFilterValue(labels[1])}{absoluteMax && value[1] >= max ? '+' : ''}
|
||||||
</span>
|
</span>
|
||||||
</div>
|
</div>
|
||||||
);
|
);
|
||||||
|
|
@ -70,19 +77,15 @@ interface FiltersProps {
|
||||||
onNavigateToSource?: (slug: string, featureName: string) => void;
|
onNavigateToSource?: (slug: string, featureName: string) => void;
|
||||||
openInfoFeature?: string | null;
|
openInfoFeature?: string | null;
|
||||||
onClearOpenInfoFeature?: () => void;
|
onClearOpenInfoFeature?: () => void;
|
||||||
travelTimeEnabled: boolean;
|
travelTimeEntries: TravelTimeEntries;
|
||||||
travelTimeDestination: [number, number] | null;
|
travelTimeDataRanges: Partial<Record<TransportMode, [number, number]>>;
|
||||||
travelTimeDestinationLabel: string;
|
onTravelTimeEnableMode: (mode: TransportMode) => void;
|
||||||
travelTimeMode: TransportMode;
|
onTravelTimeDisableMode: (mode: TransportMode) => void;
|
||||||
travelTimeRange: [number, number] | null;
|
onTravelTimeSetDestination: (mode: TransportMode, lat: number, lon: number, label: string) => void;
|
||||||
travelTimeDataRange: [number, number] | null;
|
onTravelTimeRangeChange: (mode: TransportMode, range: [number, number]) => void;
|
||||||
onTravelTimeEnable: () => void;
|
|
||||||
onTravelTimeDisable: () => void;
|
|
||||||
onTravelTimeSetDestination: (lat: number, lon: number, label: string) => void;
|
|
||||||
onTravelTimeModeChange: (mode: TransportMode) => void;
|
|
||||||
onTravelTimeRangeChange: (range: [number, number]) => void;
|
|
||||||
aiFilterLoading: boolean;
|
aiFilterLoading: boolean;
|
||||||
aiFilterError: string | null;
|
aiFilterError: string | null;
|
||||||
|
aiFilterNotes: string | null;
|
||||||
onAiFilterSubmit: (query: string) => void;
|
onAiFilterSubmit: (query: string) => void;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -104,19 +107,15 @@ export default memo(function Filters({
|
||||||
onNavigateToSource,
|
onNavigateToSource,
|
||||||
openInfoFeature,
|
openInfoFeature,
|
||||||
onClearOpenInfoFeature,
|
onClearOpenInfoFeature,
|
||||||
travelTimeEnabled,
|
travelTimeEntries,
|
||||||
travelTimeDestination,
|
travelTimeDataRanges,
|
||||||
travelTimeDestinationLabel,
|
onTravelTimeEnableMode,
|
||||||
travelTimeMode,
|
onTravelTimeDisableMode,
|
||||||
travelTimeRange,
|
|
||||||
travelTimeDataRange,
|
|
||||||
onTravelTimeEnable,
|
|
||||||
onTravelTimeDisable,
|
|
||||||
onTravelTimeSetDestination,
|
onTravelTimeSetDestination,
|
||||||
onTravelTimeModeChange,
|
|
||||||
onTravelTimeRangeChange,
|
onTravelTimeRangeChange,
|
||||||
aiFilterLoading,
|
aiFilterLoading,
|
||||||
aiFilterError,
|
aiFilterError,
|
||||||
|
aiFilterNotes,
|
||||||
onAiFilterSubmit,
|
onAiFilterSubmit,
|
||||||
}: FiltersProps) {
|
}: FiltersProps) {
|
||||||
const availableFeatures = features.filter((f) => !enabledFeatures.has(f.name));
|
const availableFeatures = features.filter((f) => !enabledFeatures.has(f.name));
|
||||||
|
|
@ -127,6 +126,11 @@ export default memo(function Filters({
|
||||||
const [activeInfoFeature, setActiveInfoFeature] = useState<FeatureMeta | null>(null);
|
const [activeInfoFeature, setActiveInfoFeature] = useState<FeatureMeta | null>(null);
|
||||||
const [collapsedGroups, toggleGroup] = useCollapsibleGroups();
|
const [collapsedGroups, toggleGroup] = useCollapsibleGroups();
|
||||||
|
|
||||||
|
const activeModes = useMemo(
|
||||||
|
() => TRANSPORT_MODES.filter((m) => m in travelTimeEntries),
|
||||||
|
[travelTimeEntries]
|
||||||
|
);
|
||||||
|
|
||||||
const handleAddAndScroll = useCallback(
|
const handleAddAndScroll = useCallback(
|
||||||
(name: string) => {
|
(name: string) => {
|
||||||
onAddFilter(name);
|
onAddFilter(name);
|
||||||
|
|
@ -144,17 +148,19 @@ export default memo(function Filters({
|
||||||
const percentileScales = useMemo(() => {
|
const percentileScales = useMemo(() => {
|
||||||
const scales = new Map<string, PercentileScale>();
|
const scales = new Map<string, PercentileScale>();
|
||||||
for (const f of features) {
|
for (const f of features) {
|
||||||
if (f.type === 'numeric' && f.histogram) {
|
if (f.type === 'numeric' && f.histogram && !f.absolute) {
|
||||||
scales.set(f.name, buildPercentileScale(f.histogram));
|
scales.set(f.name, buildPercentileScale(f.histogram));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
return scales;
|
return scales;
|
||||||
}, [features]);
|
}, [features]);
|
||||||
|
|
||||||
|
const badgeCount = enabledFeatureList.length + activeModes.length;
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<div ref={containerRef} className="flex flex-col bg-white dark:bg-navy-950 overflow-y-auto md:overflow-hidden h-full">
|
<div ref={containerRef} className="flex flex-col bg-white dark:bg-navy-950 overflow-y-auto md:overflow-hidden h-full">
|
||||||
<div className="shrink-0 border-b border-warm-200 dark:border-navy-700">
|
<div className="shrink-0 border-b border-warm-200 dark:border-navy-700">
|
||||||
<AiFilterInput loading={aiFilterLoading} error={aiFilterError} onSubmit={onAiFilterSubmit} />
|
<AiFilterInput loading={aiFilterLoading} error={aiFilterError} notes={aiFilterNotes} onSubmit={onAiFilterSubmit} />
|
||||||
<div className="flex items-center gap-2 px-3 pb-2">
|
<div className="flex items-center gap-2 px-3 pb-2">
|
||||||
<button
|
<button
|
||||||
onClick={() => setShowPhilosophy(true)}
|
onClick={() => setShowPhilosophy(true)}
|
||||||
|
|
@ -171,32 +177,34 @@ export default memo(function Filters({
|
||||||
<span className="text-sm font-semibold text-navy-950 dark:text-warm-100">
|
<span className="text-sm font-semibold text-navy-950 dark:text-warm-100">
|
||||||
Active Filters
|
Active Filters
|
||||||
</span>
|
</span>
|
||||||
{(enabledFeatureList.length > 0 || travelTimeEnabled) && (
|
{badgeCount > 0 && (
|
||||||
<span className="text-xs font-medium px-1.5 py-0.5 rounded-full bg-teal-50 dark:bg-teal-900/30 text-teal-600 dark:text-teal-400">
|
<span className="text-xs font-medium px-1.5 py-0.5 rounded-full bg-teal-50 dark:bg-teal-900/30 text-teal-600 dark:text-teal-400">
|
||||||
{enabledFeatureList.length + (travelTimeEnabled ? 1 : 0)}
|
{badgeCount}
|
||||||
</span>
|
</span>
|
||||||
)}
|
)}
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
<div className="md:flex-1 md:overflow-y-auto">
|
<div className="md:flex-1 md:overflow-y-auto">
|
||||||
{travelTimeEnabled && (
|
{activeModes.map((mode) => {
|
||||||
<div className="px-2 py-1">
|
const entry = travelTimeEntries[mode]!;
|
||||||
<TravelTimeCard
|
return (
|
||||||
destination={travelTimeDestination}
|
<div key={mode} className="px-2 py-1">
|
||||||
destinationLabel={travelTimeDestinationLabel}
|
<TravelTimeCard
|
||||||
mode={travelTimeMode}
|
mode={mode}
|
||||||
timeRange={travelTimeRange}
|
destination={entry.destination}
|
||||||
dataRange={travelTimeDataRange}
|
destinationLabel={entry.destinationLabel}
|
||||||
onSetDestination={onTravelTimeSetDestination}
|
timeRange={entry.timeRange}
|
||||||
onModeChange={onTravelTimeModeChange}
|
dataRange={travelTimeDataRanges[mode] ?? null}
|
||||||
onTimeRangeChange={onTravelTimeRangeChange}
|
onSetDestination={(lat, lon, label) => onTravelTimeSetDestination(mode, lat, lon, label)}
|
||||||
onRemove={onTravelTimeDisable}
|
onTimeRangeChange={(range) => onTravelTimeRangeChange(mode, range)}
|
||||||
/>
|
onRemove={() => onTravelTimeDisableMode(mode)}
|
||||||
</div>
|
/>
|
||||||
)}
|
</div>
|
||||||
|
);
|
||||||
|
})}
|
||||||
|
|
||||||
{enabledFeatureList.length === 0 && !travelTimeEnabled && (
|
{enabledFeatureList.length === 0 && activeModes.length === 0 && (
|
||||||
<p className="px-3 py-1.5 text-xs text-warm-400 dark:text-warm-500">
|
<p className="px-3 py-1.5 text-xs text-warm-400 dark:text-warm-500">
|
||||||
Browse features below and click + to add a filter
|
Browse features below and click + to add a filter
|
||||||
</p>
|
</p>
|
||||||
|
|
@ -300,6 +308,7 @@ export default memo(function Filters({
|
||||||
max={scale ? 100 : feature.max!}
|
max={scale ? 100 : feature.max!}
|
||||||
value={sliderValue}
|
value={sliderValue}
|
||||||
displayValues={scale ? displayValue : undefined}
|
displayValues={scale ? displayValue : undefined}
|
||||||
|
absoluteMax={feature.absolute}
|
||||||
/>
|
/>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
@ -327,8 +336,8 @@ export default memo(function Filters({
|
||||||
onNavigateToSource={onNavigateToSource}
|
onNavigateToSource={onNavigateToSource}
|
||||||
openInfoFeature={openInfoFeature}
|
openInfoFeature={openInfoFeature}
|
||||||
onClearOpenInfoFeature={onClearOpenInfoFeature}
|
onClearOpenInfoFeature={onClearOpenInfoFeature}
|
||||||
travelTimeEnabled={travelTimeEnabled}
|
activeTravelModes={activeModes}
|
||||||
onEnableTravelTime={onTravelTimeEnable}
|
onEnableTravelMode={onTravelTimeEnableMode}
|
||||||
/>
|
/>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
|
||||||
143
frontend/src/components/map/LocationSearch.tsx
Normal file
143
frontend/src/components/map/LocationSearch.tsx
Normal file
|
|
@ -0,0 +1,143 @@
|
||||||
|
import { useState, useCallback, useRef, useEffect } from 'react';
|
||||||
|
import type { PostcodeGeometry } from '../../types';
|
||||||
|
import { authHeaders } from '../../lib/api';
|
||||||
|
import { useIsMobile } from '../../hooks/useIsMobile';
|
||||||
|
import { useLocationSearch, type SearchResult } from '../../hooks/useLocationSearch';
|
||||||
|
import { PlaceSearchInput } from '../ui/PlaceSearchInput';
|
||||||
|
import { SearchIcon } from '../ui/icons/SearchIcon';
|
||||||
|
|
||||||
|
export interface SearchedLocation {
|
||||||
|
postcode: string;
|
||||||
|
geometry: PostcodeGeometry;
|
||||||
|
}
|
||||||
|
|
||||||
|
const ZOOM_FOR_TYPE: Record<string, number> = {
|
||||||
|
city: 10,
|
||||||
|
borough: 12,
|
||||||
|
town: 13,
|
||||||
|
suburb: 14,
|
||||||
|
quarter: 14,
|
||||||
|
neighbourhood: 14,
|
||||||
|
village: 14,
|
||||||
|
station: 15,
|
||||||
|
island: 12,
|
||||||
|
locality: 14,
|
||||||
|
hamlet: 15,
|
||||||
|
isolated_dwelling: 16,
|
||||||
|
};
|
||||||
|
|
||||||
|
export default function LocationSearch({
|
||||||
|
onFlyTo,
|
||||||
|
onLocationSearched,
|
||||||
|
}: {
|
||||||
|
onFlyTo: (lat: number, lng: number, zoom: number) => void;
|
||||||
|
onLocationSearched?: (postcode: SearchedLocation | null) => void;
|
||||||
|
}) {
|
||||||
|
const search = useLocationSearch();
|
||||||
|
const [error, setError] = useState<string | null>(null);
|
||||||
|
const [loading, setLoading] = useState(false);
|
||||||
|
const [expanded, setExpanded] = useState(false);
|
||||||
|
const isMobile = useIsMobile();
|
||||||
|
const containerRef = useRef<HTMLDivElement>(null);
|
||||||
|
const inputRef = useRef<HTMLInputElement>(null);
|
||||||
|
|
||||||
|
// Close on outside click
|
||||||
|
useEffect(() => {
|
||||||
|
const handler = (e: MouseEvent) => {
|
||||||
|
if (containerRef.current && !containerRef.current.contains(e.target as Node)) {
|
||||||
|
search.close();
|
||||||
|
if (isMobile) setExpanded(false);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
document.addEventListener('mousedown', handler);
|
||||||
|
return () => document.removeEventListener('mousedown', handler);
|
||||||
|
}, [isMobile, search]);
|
||||||
|
|
||||||
|
// Focus input when expanding on mobile
|
||||||
|
useEffect(() => {
|
||||||
|
if (isMobile && expanded) {
|
||||||
|
inputRef.current?.focus();
|
||||||
|
}
|
||||||
|
}, [isMobile, expanded]);
|
||||||
|
|
||||||
|
const selectResult = useCallback(
|
||||||
|
async (result: SearchResult) => {
|
||||||
|
if (result.type === 'place') {
|
||||||
|
const zoom = ZOOM_FOR_TYPE[result.place_type] ?? 14;
|
||||||
|
onFlyTo(result.lat, result.lon, zoom);
|
||||||
|
onLocationSearched?.(null);
|
||||||
|
search.clear();
|
||||||
|
if (isMobile) setExpanded(false);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Postcode — fetch geometry
|
||||||
|
setError(null);
|
||||||
|
setLoading(true);
|
||||||
|
search.close();
|
||||||
|
try {
|
||||||
|
const res = await fetch(
|
||||||
|
`/api/postcode/${encodeURIComponent(result.label)}`,
|
||||||
|
authHeaders(),
|
||||||
|
);
|
||||||
|
if (!res.ok) {
|
||||||
|
setError('Postcode not found');
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
const json: {
|
||||||
|
postcode: string;
|
||||||
|
latitude: number;
|
||||||
|
longitude: number;
|
||||||
|
geometry: PostcodeGeometry;
|
||||||
|
} = await res.json();
|
||||||
|
onFlyTo(json.latitude, json.longitude, 16);
|
||||||
|
onLocationSearched?.({ postcode: json.postcode, geometry: json.geometry });
|
||||||
|
search.clear();
|
||||||
|
if (isMobile) setExpanded(false);
|
||||||
|
} catch {
|
||||||
|
setError('Lookup failed');
|
||||||
|
} finally {
|
||||||
|
setLoading(false);
|
||||||
|
}
|
||||||
|
},
|
||||||
|
[onFlyTo, onLocationSearched, isMobile, search],
|
||||||
|
);
|
||||||
|
|
||||||
|
// Mobile collapsed state: just a search icon button
|
||||||
|
if (isMobile && !expanded) {
|
||||||
|
return (
|
||||||
|
<button
|
||||||
|
type="button"
|
||||||
|
onClick={() => setExpanded(true)}
|
||||||
|
className="absolute top-3 left-3 z-10 p-2 bg-white dark:bg-warm-800 rounded shadow-lg"
|
||||||
|
aria-label="Search places or postcodes"
|
||||||
|
>
|
||||||
|
<SearchIcon className="w-5 h-5 text-warm-600 dark:text-warm-300" />
|
||||||
|
</button>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div ref={containerRef} className="absolute top-3 left-3 z-10 flex flex-col">
|
||||||
|
<div className="flex items-center shadow-lg rounded overflow-hidden bg-white dark:bg-warm-800">
|
||||||
|
<SearchIcon className="w-4 h-4 text-warm-400 dark:text-warm-500 ml-3 shrink-0" />
|
||||||
|
<PlaceSearchInput
|
||||||
|
search={search}
|
||||||
|
onSelect={selectResult}
|
||||||
|
loading={loading}
|
||||||
|
placeholder="Search places or postcodes..."
|
||||||
|
size="sm"
|
||||||
|
inputClassName="px-2 py-2 text-sm w-56 border-none outline-none bg-transparent text-warm-700 dark:text-warm-200 placeholder-warm-400 dark:placeholder-warm-500"
|
||||||
|
inputRef={inputRef}
|
||||||
|
onInputChange={() => setError(null)}
|
||||||
|
/>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
{error && (
|
||||||
|
<span className="text-xs text-red-600 dark:text-red-300 bg-white/90 dark:bg-warm-800/90 rounded px-2 py-0.5 shadow mt-1">
|
||||||
|
{error}
|
||||||
|
</span>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
@ -6,6 +6,7 @@ import 'maplibre-gl/dist/maplibre-gl.css';
|
||||||
import type {
|
import type {
|
||||||
HexagonData,
|
HexagonData,
|
||||||
PostcodeFeature,
|
PostcodeFeature,
|
||||||
|
PostcodeGeometry,
|
||||||
ViewState,
|
ViewState,
|
||||||
ViewChangeParams,
|
ViewChangeParams,
|
||||||
POI,
|
POI,
|
||||||
|
|
@ -15,11 +16,12 @@ import type {
|
||||||
|
|
||||||
import { zoomToResolution, getBoundsFromViewState, getMapStyle } from '../../lib/map-utils';
|
import { zoomToResolution, getBoundsFromViewState, getMapStyle } from '../../lib/map-utils';
|
||||||
import { INITIAL_VIEW_STATE, MAP_MIN_ZOOM, MAP_BOUNDS } from '../../lib/consts';
|
import { INITIAL_VIEW_STATE, MAP_MIN_ZOOM, MAP_BOUNDS } from '../../lib/consts';
|
||||||
import PostcodeSearch, { type SearchedPostcode } from './PostcodeSearch';
|
import LocationSearch, { type SearchedLocation } from './LocationSearch';
|
||||||
import MapLegend from './MapLegend';
|
import MapLegend from './MapLegend';
|
||||||
import HoverCard from './HoverCard';
|
import HoverCard from './HoverCard';
|
||||||
import type { FeatureFilters } from '../../types';
|
import type { FeatureFilters } from '../../types';
|
||||||
import { useDeckLayers, osmIdToUrl } from '../../hooks/useDeckLayers';
|
import { useDeckLayers, osmIdToUrl } from '../../hooks/useDeckLayers';
|
||||||
|
import { MODE_LABELS, type TransportMode, type TravelTimeEntries } from '../../hooks/useTravelTime';
|
||||||
|
|
||||||
interface MapProps {
|
interface MapProps {
|
||||||
data: HexagonData[];
|
data: HexagonData[];
|
||||||
|
|
@ -42,14 +44,12 @@ interface MapProps {
|
||||||
screenshotMode?: boolean;
|
screenshotMode?: boolean;
|
||||||
ogMode?: boolean;
|
ogMode?: boolean;
|
||||||
filters?: FeatureFilters;
|
filters?: FeatureFilters;
|
||||||
searchedPostcode?: SearchedPostcode | null;
|
selectedPostcodeGeometry?: PostcodeGeometry | null;
|
||||||
onPostcodeSearched?: (postcode: SearchedPostcode | null) => void;
|
onLocationSearched?: (location: SearchedLocation | null) => void;
|
||||||
bounds?: Bounds | null;
|
bounds?: Bounds | null;
|
||||||
hideLegend?: boolean;
|
hideLegend?: boolean;
|
||||||
travelTimeEnabled?: boolean;
|
travelTimeEntries?: TravelTimeEntries;
|
||||||
travelTimeDestination?: [number, number] | null;
|
travelTimeColorRanges?: Partial<Record<TransportMode, [number, number]>>;
|
||||||
travelTimeColorRange?: [number, number] | null;
|
|
||||||
travelTimeRange?: [number, number] | null;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
interface Dimensions {
|
interface Dimensions {
|
||||||
|
|
@ -98,14 +98,12 @@ export default memo(function Map({
|
||||||
screenshotMode = false,
|
screenshotMode = false,
|
||||||
ogMode = false,
|
ogMode = false,
|
||||||
filters = {},
|
filters = {},
|
||||||
searchedPostcode,
|
selectedPostcodeGeometry,
|
||||||
onPostcodeSearched,
|
onLocationSearched,
|
||||||
bounds: viewportBounds,
|
bounds: viewportBounds,
|
||||||
hideLegend = false,
|
hideLegend = false,
|
||||||
travelTimeEnabled = false,
|
travelTimeEntries = {},
|
||||||
travelTimeDestination,
|
travelTimeColorRanges = {},
|
||||||
travelTimeColorRange,
|
|
||||||
travelTimeRange,
|
|
||||||
}: MapProps) {
|
}: MapProps) {
|
||||||
const containerRef = useRef<HTMLDivElement>(null);
|
const containerRef = useRef<HTMLDivElement>(null);
|
||||||
const [viewState, setViewState] = useState<ViewState>(initialViewState || INITIAL_VIEW_STATE);
|
const [viewState, setViewState] = useState<ViewState>(initialViewState || INITIAL_VIEW_STATE);
|
||||||
|
|
@ -168,6 +166,7 @@ export default memo(function Map({
|
||||||
postcodeCountRange,
|
postcodeCountRange,
|
||||||
colorFeatureMeta,
|
colorFeatureMeta,
|
||||||
handleMouseLeave,
|
handleMouseLeave,
|
||||||
|
primaryTravelMode,
|
||||||
} = useDeckLayers({
|
} = useDeckLayers({
|
||||||
data,
|
data,
|
||||||
postcodeData,
|
postcodeData,
|
||||||
|
|
@ -182,12 +181,10 @@ export default memo(function Map({
|
||||||
onHexagonClick,
|
onHexagonClick,
|
||||||
onHexagonHover,
|
onHexagonHover,
|
||||||
theme,
|
theme,
|
||||||
searchedPostcode,
|
selectedPostcodeGeometry,
|
||||||
bounds: viewportBounds,
|
bounds: viewportBounds,
|
||||||
travelTimeEnabled,
|
travelTimeEntries,
|
||||||
travelTimeDestination,
|
travelTimeColorRanges,
|
||||||
travelTimeColorRange,
|
|
||||||
travelTimeRange,
|
|
||||||
});
|
});
|
||||||
|
|
||||||
return (
|
return (
|
||||||
|
|
@ -222,12 +219,12 @@ export default memo(function Map({
|
||||||
) : null
|
) : null
|
||||||
) : (
|
) : (
|
||||||
<>
|
<>
|
||||||
<PostcodeSearch onFlyTo={handleFlyTo} onPostcodeSearched={onPostcodeSearched} />
|
<LocationSearch onFlyTo={handleFlyTo} onLocationSearched={onLocationSearched} />
|
||||||
{!hideLegend &&
|
{!hideLegend &&
|
||||||
(travelTimeEnabled && travelTimeDestination && travelTimeColorRange ? (
|
(primaryTravelMode && travelTimeColorRanges[primaryTravelMode] ? (
|
||||||
<MapLegend
|
<MapLegend
|
||||||
featureLabel="Travel time"
|
featureLabel={`Travel time (${MODE_LABELS[primaryTravelMode]})`}
|
||||||
range={travelTimeColorRange}
|
range={travelTimeColorRanges[primaryTravelMode]!}
|
||||||
showCancel={false}
|
showCancel={false}
|
||||||
onCancel={onCancelPin}
|
onCancel={onCancelPin}
|
||||||
mode="feature"
|
mode="feature"
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
import { useState, useEffect, useMemo, useCallback } from 'react';
|
import { useState, useEffect, useMemo, useCallback } from 'react';
|
||||||
import type { FeatureMeta, FeatureFilters, POICategoryGroup, ViewState } from '../../types';
|
import type { FeatureMeta, FeatureFilters, POICategoryGroup, ViewState } from '../../types';
|
||||||
import type { SearchedPostcode } from './PostcodeSearch';
|
import type { SearchedLocation } from './LocationSearch';
|
||||||
import type { Page } from '../ui/Header';
|
import type { Page } from '../ui/Header';
|
||||||
import Map from './Map';
|
import Map from './Map';
|
||||||
import Filters from './Filters';
|
import Filters from './Filters';
|
||||||
|
|
@ -18,8 +18,14 @@ import { usePaneResize } from '../../hooks/usePaneResize';
|
||||||
import { useAiFilters } from '../../hooks/useAiFilters';
|
import { useAiFilters } from '../../hooks/useAiFilters';
|
||||||
import { useAreaSummary } from '../../hooks/useAreaSummary';
|
import { useAreaSummary } from '../../hooks/useAreaSummary';
|
||||||
import { useUrlSync } from '../../hooks/useUrlSync';
|
import { useUrlSync } from '../../hooks/useUrlSync';
|
||||||
import { useTravelTime, type TravelTimeInitial } from '../../hooks/useTravelTime';
|
import {
|
||||||
import { apiUrl, buildFilterString } from '../../lib/api';
|
useTravelTime,
|
||||||
|
TRANSPORT_MODES,
|
||||||
|
MODE_LABELS,
|
||||||
|
type TransportMode,
|
||||||
|
type TravelTimeInitial,
|
||||||
|
} from '../../hooks/useTravelTime';
|
||||||
|
import { apiUrl, assertOk, buildFilterString, logNonAbortError } from '../../lib/api';
|
||||||
import { SpinnerIcon } from '../ui/icons/SpinnerIcon';
|
import { SpinnerIcon } from '../ui/icons/SpinnerIcon';
|
||||||
import { MapPinIcon } from '../ui/icons/MapPinIcon';
|
import { MapPinIcon } from '../ui/icons/MapPinIcon';
|
||||||
|
|
||||||
|
|
@ -65,7 +71,6 @@ export default function MapPage({
|
||||||
isMobile = false,
|
isMobile = false,
|
||||||
initialTravelTime,
|
initialTravelTime,
|
||||||
}: MapPageProps) {
|
}: MapPageProps) {
|
||||||
const [searchedPostcode, setSearchedPostcode] = useState<SearchedPostcode | null>(null);
|
|
||||||
const [selectedPOICategories, setSelectedPOICategories] =
|
const [selectedPOICategories, setSelectedPOICategories] =
|
||||||
useState<Set<string>>(initialPOICategories);
|
useState<Set<string>>(initialPOICategories);
|
||||||
|
|
||||||
|
|
@ -109,7 +114,7 @@ export default function MapPage({
|
||||||
const handleAiFilterSubmit = useCallback(
|
const handleAiFilterSubmit = useCallback(
|
||||||
async (query: string) => {
|
async (query: string) => {
|
||||||
const result = await aiFilters.fetchAiFilters(query);
|
const result = await aiFilters.fetchAiFilters(query);
|
||||||
if (result) handleSetFilters(result);
|
if (result) handleSetFilters(result.filters);
|
||||||
},
|
},
|
||||||
[aiFilters.fetchAiFilters, handleSetFilters]
|
[aiFilters.fetchAiFilters, handleSetFilters]
|
||||||
);
|
);
|
||||||
|
|
@ -125,9 +130,7 @@ export default function MapPage({
|
||||||
activeFeature,
|
activeFeature,
|
||||||
dragValue,
|
dragValue,
|
||||||
dragData,
|
dragData,
|
||||||
travelTimeEnabled: travelTime.enabled,
|
travelTimeEntries: travelTime.entries,
|
||||||
travelTimeDestination: travelTime.destination,
|
|
||||||
travelTimeMode: travelTime.mode,
|
|
||||||
});
|
});
|
||||||
|
|
||||||
// Keep filter bounds in sync with map data
|
// Keep filter bounds in sync with map data
|
||||||
|
|
@ -142,24 +145,42 @@ export default function MapPage({
|
||||||
resolution: mapData.resolution,
|
resolution: mapData.resolution,
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// Location search handler — selects postcode + shows stats
|
||||||
|
const handleLocationSearchResult = useCallback(
|
||||||
|
(result: SearchedLocation | null) => {
|
||||||
|
if (result) {
|
||||||
|
selection.handleLocationSearch(result.postcode, result.geometry);
|
||||||
|
if (isMobile) setMobileDrawerOpen(true);
|
||||||
|
} else {
|
||||||
|
selection.handleCloseSelection();
|
||||||
|
}
|
||||||
|
},
|
||||||
|
[selection.handleLocationSearch, selection.handleCloseSelection, isMobile]
|
||||||
|
);
|
||||||
|
|
||||||
// POI data
|
// POI data
|
||||||
const pois = usePOIData(mapData.bounds, selectedPOICategories);
|
const pois = usePOIData(mapData.bounds, selectedPOICategories);
|
||||||
|
|
||||||
// Compute data range for travel time slider
|
// Compute data range for travel time slider per mode (full min/max for slider bounds)
|
||||||
const travelTimeDataRange = useMemo((): [number, number] | null => {
|
const travelTimeDataRanges = useMemo((): Partial<Record<TransportMode, [number, number]>> => {
|
||||||
if (!travelTime.enabled || !travelTime.destination) return null;
|
const ranges: Partial<Record<TransportMode, [number, number]>> = {};
|
||||||
const vals: number[] = [];
|
for (const mode of TRANSPORT_MODES) {
|
||||||
for (const item of mapData.data) {
|
const entry = travelTime.entries[mode];
|
||||||
const val = item.travel_time;
|
if (!entry?.destination) continue;
|
||||||
if (typeof val === 'number' && !isNaN(val)) vals.push(val);
|
const vals: number[] = [];
|
||||||
|
for (const item of mapData.data) {
|
||||||
|
const val = item[`travel_time_${mode}`];
|
||||||
|
if (typeof val === 'number' && !isNaN(val)) vals.push(val);
|
||||||
|
}
|
||||||
|
if (vals.length === 0) continue;
|
||||||
|
vals.sort((a, b) => a - b);
|
||||||
|
ranges[mode] = [vals[0], vals[vals.length - 1]];
|
||||||
}
|
}
|
||||||
if (vals.length === 0) return null;
|
return ranges;
|
||||||
vals.sort((a, b) => a - b);
|
}, [travelTime.entries, mapData.data]);
|
||||||
return [vals[0], vals[vals.length - 1]];
|
|
||||||
}, [travelTime.enabled, travelTime.destination, mapData.data]);
|
|
||||||
|
|
||||||
// Sync current state to URL
|
// Sync current state to URL
|
||||||
useUrlSync(mapData.currentView, filters, features, selectedPOICategories, selection.rightPaneTab, travelTime);
|
useUrlSync(mapData.currentView, filters, features, selectedPOICategories, selection.rightPaneTab, travelTime.entries);
|
||||||
|
|
||||||
// Set initial view and tab from URL state
|
// Set initial view and tab from URL state
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
|
|
@ -238,7 +259,7 @@ export default function MapPage({
|
||||||
link.click();
|
link.click();
|
||||||
URL.revokeObjectURL(link.href);
|
URL.revokeObjectURL(link.href);
|
||||||
})
|
})
|
||||||
.catch((err) => console.error('Export failed:', err))
|
.catch((err) => logNonAbortError('Export failed', err))
|
||||||
.finally(() => setExporting(false));
|
.finally(() => setExporting(false));
|
||||||
}, [mapData.bounds, filters, features, exporting]);
|
}, [mapData.bounds, filters, features, exporting]);
|
||||||
|
|
||||||
|
|
@ -258,10 +279,7 @@ export default function MapPage({
|
||||||
let min = Infinity;
|
let min = Infinity;
|
||||||
let max = -Infinity;
|
let max = -Infinity;
|
||||||
for (const d of items) {
|
for (const d of items) {
|
||||||
const c =
|
const c = 'count' in d ? d.count : d.properties.count;
|
||||||
'count' in d
|
|
||||||
? (d as { count: number }).count
|
|
||||||
: (d as { properties: { count: number } }).properties.count;
|
|
||||||
if (c < min) min = c;
|
if (c < min) min = c;
|
||||||
if (c > max) max = c;
|
if (c > max) max = c;
|
||||||
}
|
}
|
||||||
|
|
@ -301,10 +319,8 @@ export default function MapPage({
|
||||||
screenshotMode
|
screenshotMode
|
||||||
ogMode={ogMode}
|
ogMode={ogMode}
|
||||||
bounds={mapData.bounds}
|
bounds={mapData.bounds}
|
||||||
travelTimeEnabled={travelTime.enabled}
|
travelTimeEntries={travelTime.entries}
|
||||||
travelTimeDestination={travelTime.destination}
|
travelTimeColorRanges={mapData.travelTimeColorRanges}
|
||||||
travelTimeColorRange={mapData.travelTimeColorRange}
|
|
||||||
travelTimeRange={travelTime.timeRange}
|
|
||||||
/>
|
/>
|
||||||
</div>
|
</div>
|
||||||
);
|
);
|
||||||
|
|
@ -373,19 +389,15 @@ export default function MapPage({
|
||||||
onCancelPin={handleCancelPin}
|
onCancelPin={handleCancelPin}
|
||||||
openInfoFeature={pendingInfoFeature}
|
openInfoFeature={pendingInfoFeature}
|
||||||
onClearOpenInfoFeature={onClearPendingInfoFeature}
|
onClearOpenInfoFeature={onClearPendingInfoFeature}
|
||||||
travelTimeEnabled={travelTime.enabled}
|
travelTimeEntries={travelTime.entries}
|
||||||
travelTimeDestination={travelTime.destination}
|
travelTimeDataRanges={travelTimeDataRanges}
|
||||||
travelTimeDestinationLabel={travelTime.destinationLabel}
|
onTravelTimeEnableMode={travelTime.handleEnableMode}
|
||||||
travelTimeMode={travelTime.mode}
|
onTravelTimeDisableMode={travelTime.handleDisableMode}
|
||||||
travelTimeRange={travelTime.timeRange}
|
|
||||||
travelTimeDataRange={travelTimeDataRange}
|
|
||||||
onTravelTimeEnable={travelTime.handleEnable}
|
|
||||||
onTravelTimeDisable={travelTime.handleDisable}
|
|
||||||
onTravelTimeSetDestination={travelTime.handleSetDestination}
|
onTravelTimeSetDestination={travelTime.handleSetDestination}
|
||||||
onTravelTimeModeChange={travelTime.handleModeChange}
|
|
||||||
onTravelTimeRangeChange={travelTime.handleTimeRangeChange}
|
onTravelTimeRangeChange={travelTime.handleTimeRangeChange}
|
||||||
aiFilterLoading={aiFilters.loading}
|
aiFilterLoading={aiFilters.loading}
|
||||||
aiFilterError={aiFilters.error}
|
aiFilterError={aiFilters.error}
|
||||||
|
aiFilterNotes={aiFilters.notes}
|
||||||
onAiFilterSubmit={handleAiFilterSubmit}
|
onAiFilterSubmit={handleAiFilterSubmit}
|
||||||
/>
|
/>
|
||||||
);
|
);
|
||||||
|
|
@ -426,14 +438,12 @@ export default function MapPage({
|
||||||
initialViewState={initialViewState}
|
initialViewState={initialViewState}
|
||||||
theme={theme}
|
theme={theme}
|
||||||
filters={filters}
|
filters={filters}
|
||||||
searchedPostcode={searchedPostcode}
|
selectedPostcodeGeometry={selection.selectedPostcodeGeometry}
|
||||||
onPostcodeSearched={setSearchedPostcode}
|
onLocationSearched={handleLocationSearchResult}
|
||||||
bounds={mapData.bounds}
|
bounds={mapData.bounds}
|
||||||
hideLegend
|
hideLegend
|
||||||
travelTimeEnabled={travelTime.enabled}
|
travelTimeEntries={travelTime.entries}
|
||||||
travelTimeDestination={travelTime.destination}
|
travelTimeColorRanges={mapData.travelTimeColorRanges}
|
||||||
travelTimeColorRange={mapData.travelTimeColorRange}
|
|
||||||
travelTimeRange={travelTime.timeRange}
|
|
||||||
/>
|
/>
|
||||||
{mapData.loading && (
|
{mapData.loading && (
|
||||||
<div className="absolute bottom-2 left-2 bg-white dark:bg-navy-800 dark:text-warm-200 px-2 py-1 rounded shadow text-xs">
|
<div className="absolute bottom-2 left-2 bg-white dark:bg-navy-800 dark:text-warm-200 px-2 py-1 rounded shadow text-xs">
|
||||||
|
|
@ -461,43 +471,54 @@ export default function MapPage({
|
||||||
style={{ flex: '55 0 0' }}
|
style={{ flex: '55 0 0' }}
|
||||||
>
|
>
|
||||||
{/* Legend */}
|
{/* Legend */}
|
||||||
{travelTime.enabled && travelTime.destination && mapData.travelTimeColorRange ? (
|
{(() => {
|
||||||
<MapLegend
|
const primaryMode = TRANSPORT_MODES.find(
|
||||||
featureLabel="Travel time"
|
(m) => travelTime.entries[m]?.destination && mapData.travelTimeColorRanges[m]
|
||||||
range={mapData.travelTimeColorRange}
|
);
|
||||||
showCancel={false}
|
if (primaryMode) {
|
||||||
onCancel={handleCancelPin}
|
return (
|
||||||
mode="feature"
|
<MapLegend
|
||||||
theme={theme}
|
featureLabel={`Travel time (${MODE_LABELS[primaryMode]})`}
|
||||||
inline
|
range={mapData.travelTimeColorRanges[primaryMode]!}
|
||||||
suffix=" min"
|
showCancel={false}
|
||||||
/>
|
onCancel={handleCancelPin}
|
||||||
) : viewFeature && mapData.colorRange && mobileLegendMeta ? (
|
mode="feature"
|
||||||
<MapLegend
|
theme={theme}
|
||||||
featureLabel={
|
inline
|
||||||
viewSource === 'eye'
|
suffix=" min"
|
||||||
? `Previewing \u201c${mobileLegendMeta.name}\u201d`
|
/>
|
||||||
: mobileLegendMeta.name
|
);
|
||||||
}
|
}
|
||||||
range={mapData.colorRange}
|
if (viewFeature && mapData.colorRange && mobileLegendMeta) {
|
||||||
showCancel={viewSource === 'eye'}
|
return (
|
||||||
onCancel={handleCancelPin}
|
<MapLegend
|
||||||
mode="feature"
|
featureLabel={
|
||||||
enumValues={mobileLegendMeta.type === 'enum' ? mobileLegendMeta.values : undefined}
|
viewSource === 'eye'
|
||||||
theme={theme}
|
? `Previewing \u201c${mobileLegendMeta.name}\u201d`
|
||||||
inline
|
: mobileLegendMeta.name
|
||||||
/>
|
}
|
||||||
) : (
|
range={mapData.colorRange}
|
||||||
<MapLegend
|
showCancel={viewSource === 'eye'}
|
||||||
featureLabel="Property density"
|
onCancel={handleCancelPin}
|
||||||
range={mobileDensityRange}
|
mode="feature"
|
||||||
showCancel={false}
|
enumValues={mobileLegendMeta.type === 'enum' ? mobileLegendMeta.values : undefined}
|
||||||
onCancel={handleCancelPin}
|
theme={theme}
|
||||||
mode="density"
|
inline
|
||||||
theme={theme}
|
/>
|
||||||
inline
|
);
|
||||||
/>
|
}
|
||||||
)}
|
return (
|
||||||
|
<MapLegend
|
||||||
|
featureLabel="Property density"
|
||||||
|
range={mobileDensityRange}
|
||||||
|
showCancel={false}
|
||||||
|
onCancel={handleCancelPin}
|
||||||
|
mode="density"
|
||||||
|
theme={theme}
|
||||||
|
inline
|
||||||
|
/>
|
||||||
|
);
|
||||||
|
})()}
|
||||||
{/* Filters content */}
|
{/* Filters content */}
|
||||||
<div className="flex-1 min-h-0">
|
<div className="flex-1 min-h-0">
|
||||||
{renderFilters()}
|
{renderFilters()}
|
||||||
|
|
@ -565,13 +586,11 @@ export default function MapPage({
|
||||||
initialViewState={initialViewState}
|
initialViewState={initialViewState}
|
||||||
theme={theme}
|
theme={theme}
|
||||||
filters={filters}
|
filters={filters}
|
||||||
searchedPostcode={searchedPostcode}
|
selectedPostcodeGeometry={selection.selectedPostcodeGeometry}
|
||||||
onPostcodeSearched={setSearchedPostcode}
|
onLocationSearched={handleLocationSearchResult}
|
||||||
bounds={mapData.bounds}
|
bounds={mapData.bounds}
|
||||||
travelTimeEnabled={travelTime.enabled}
|
travelTimeEntries={travelTime.entries}
|
||||||
travelTimeDestination={travelTime.destination}
|
travelTimeColorRanges={mapData.travelTimeColorRanges}
|
||||||
travelTimeColorRange={mapData.travelTimeColorRange}
|
|
||||||
travelTimeRange={travelTime.timeRange}
|
|
||||||
/>
|
/>
|
||||||
{mapData.loading && (
|
{mapData.loading && (
|
||||||
<div className="absolute bottom-4 left-4 bg-white dark:bg-navy-800 dark:text-warm-200 px-3 py-1 rounded shadow">
|
<div className="absolute bottom-4 left-4 bg-white dark:bg-navy-800 dark:text-warm-200 px-3 py-1 rounded shadow">
|
||||||
|
|
|
||||||
|
|
@ -1,300 +0,0 @@
|
||||||
import { useState, useCallback, useRef, useEffect } from 'react';
|
|
||||||
import type { PostcodeGeometry, PlaceResult } from '../../types';
|
|
||||||
import { authHeaders, logNonAbortError } from '../../lib/api';
|
|
||||||
import { useIsMobile } from '../../hooks/useIsMobile';
|
|
||||||
import { SearchIcon } from '../ui/icons/SearchIcon';
|
|
||||||
import { MapPinIcon } from '../ui/icons/MapPinIcon';
|
|
||||||
|
|
||||||
export interface SearchedPostcode {
|
|
||||||
postcode: string;
|
|
||||||
geometry: PostcodeGeometry;
|
|
||||||
}
|
|
||||||
|
|
||||||
const POSTCODE_RE = /^[A-Z]{1,2}\d[A-Z\d]?\s*\d?[A-Z]{0,2}$/i;
|
|
||||||
function looksLikePostcode(s: string) {
|
|
||||||
return POSTCODE_RE.test(s.trim());
|
|
||||||
}
|
|
||||||
|
|
||||||
type SearchResult =
|
|
||||||
| { type: 'postcode'; label: string }
|
|
||||||
| { type: 'place'; name: string; place_type: string; lat: number; lon: number };
|
|
||||||
|
|
||||||
const ZOOM_FOR_TYPE: Record<string, number> = {
|
|
||||||
city: 10,
|
|
||||||
borough: 12,
|
|
||||||
town: 13,
|
|
||||||
suburb: 14,
|
|
||||||
neighbourhood: 14,
|
|
||||||
village: 14,
|
|
||||||
locality: 14,
|
|
||||||
hamlet: 15,
|
|
||||||
isolated_dwelling: 16,
|
|
||||||
};
|
|
||||||
|
|
||||||
export default function PostcodeSearch({
|
|
||||||
onFlyTo,
|
|
||||||
onPostcodeSearched,
|
|
||||||
}: {
|
|
||||||
onFlyTo: (lat: number, lng: number, zoom: number) => void;
|
|
||||||
onPostcodeSearched?: (postcode: SearchedPostcode | null) => void;
|
|
||||||
}) {
|
|
||||||
const [query, setQuery] = useState('');
|
|
||||||
const [results, setResults] = useState<SearchResult[]>([]);
|
|
||||||
const [activeIndex, setActiveIndex] = useState(-1);
|
|
||||||
const [open, setOpen] = useState(false);
|
|
||||||
const [error, setError] = useState<string | null>(null);
|
|
||||||
const [loading, setLoading] = useState(false);
|
|
||||||
const [expanded, setExpanded] = useState(false);
|
|
||||||
const isMobile = useIsMobile();
|
|
||||||
const containerRef = useRef<HTMLDivElement>(null);
|
|
||||||
const inputRef = useRef<HTMLInputElement>(null);
|
|
||||||
const abortRef = useRef<AbortController | null>(null);
|
|
||||||
const debounceRef = useRef<ReturnType<typeof setTimeout>>();
|
|
||||||
|
|
||||||
// Close on outside click
|
|
||||||
useEffect(() => {
|
|
||||||
const handler = (e: MouseEvent) => {
|
|
||||||
if (containerRef.current && !containerRef.current.contains(e.target as Node)) {
|
|
||||||
setOpen(false);
|
|
||||||
if (isMobile) setExpanded(false);
|
|
||||||
}
|
|
||||||
};
|
|
||||||
document.addEventListener('mousedown', handler);
|
|
||||||
return () => document.removeEventListener('mousedown', handler);
|
|
||||||
}, [isMobile]);
|
|
||||||
|
|
||||||
// Focus input when expanding on mobile
|
|
||||||
useEffect(() => {
|
|
||||||
if (isMobile && expanded) {
|
|
||||||
inputRef.current?.focus();
|
|
||||||
}
|
|
||||||
}, [isMobile, expanded]);
|
|
||||||
|
|
||||||
const selectPostcode = useCallback(
|
|
||||||
async (postcode: string) => {
|
|
||||||
setError(null);
|
|
||||||
setLoading(true);
|
|
||||||
setOpen(false);
|
|
||||||
try {
|
|
||||||
const res = await fetch(
|
|
||||||
`/api/postcode/${encodeURIComponent(postcode.trim())}`,
|
|
||||||
authHeaders()
|
|
||||||
);
|
|
||||||
if (!res.ok) {
|
|
||||||
setError('Postcode not found');
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
const json: {
|
|
||||||
postcode: string;
|
|
||||||
latitude: number;
|
|
||||||
longitude: number;
|
|
||||||
geometry: PostcodeGeometry;
|
|
||||||
} = await res.json();
|
|
||||||
onFlyTo(json.latitude, json.longitude, 16);
|
|
||||||
onPostcodeSearched?.({ postcode: json.postcode, geometry: json.geometry });
|
|
||||||
setQuery('');
|
|
||||||
setResults([]);
|
|
||||||
if (isMobile) setExpanded(false);
|
|
||||||
} catch {
|
|
||||||
setError('Lookup failed');
|
|
||||||
} finally {
|
|
||||||
setLoading(false);
|
|
||||||
}
|
|
||||||
},
|
|
||||||
[onFlyTo, onPostcodeSearched, isMobile]
|
|
||||||
);
|
|
||||||
|
|
||||||
const selectPlace = useCallback(
|
|
||||||
(place: { name: string; place_type: string; lat: number; lon: number }) => {
|
|
||||||
const zoom = ZOOM_FOR_TYPE[place.place_type] ?? 14;
|
|
||||||
onFlyTo(place.lat, place.lon, zoom);
|
|
||||||
setQuery('');
|
|
||||||
setResults([]);
|
|
||||||
setOpen(false);
|
|
||||||
if (isMobile) setExpanded(false);
|
|
||||||
},
|
|
||||||
[onFlyTo, isMobile]
|
|
||||||
);
|
|
||||||
|
|
||||||
const selectResult = useCallback(
|
|
||||||
(result: SearchResult) => {
|
|
||||||
if (result.type === 'postcode') {
|
|
||||||
selectPostcode(result.label);
|
|
||||||
} else {
|
|
||||||
selectPlace(result);
|
|
||||||
}
|
|
||||||
},
|
|
||||||
[selectPostcode, selectPlace]
|
|
||||||
);
|
|
||||||
|
|
||||||
const handleInputChange = useCallback((value: string) => {
|
|
||||||
setQuery(value);
|
|
||||||
setError(null);
|
|
||||||
setActiveIndex(-1);
|
|
||||||
|
|
||||||
// Cancel in-flight request
|
|
||||||
abortRef.current?.abort();
|
|
||||||
if (debounceRef.current) clearTimeout(debounceRef.current);
|
|
||||||
|
|
||||||
const trimmed = value.trim();
|
|
||||||
if (!trimmed) {
|
|
||||||
setResults([]);
|
|
||||||
setOpen(false);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (looksLikePostcode(trimmed)) {
|
|
||||||
setResults([{ type: 'postcode', label: trimmed.toUpperCase() }]);
|
|
||||||
setOpen(true);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (trimmed.length < 2) {
|
|
||||||
setResults([]);
|
|
||||||
setOpen(false);
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
// Debounced place search
|
|
||||||
debounceRef.current = setTimeout(async () => {
|
|
||||||
const controller = new AbortController();
|
|
||||||
abortRef.current = controller;
|
|
||||||
try {
|
|
||||||
const params = new URLSearchParams({ q: trimmed, limit: '7' });
|
|
||||||
const res = await fetch(
|
|
||||||
`/api/places?${params}`,
|
|
||||||
authHeaders({ signal: controller.signal })
|
|
||||||
);
|
|
||||||
if (!res.ok) return;
|
|
||||||
const json: { places: PlaceResult[] } = await res.json();
|
|
||||||
const placeResults: SearchResult[] = json.places.map((p) => ({
|
|
||||||
type: 'place' as const,
|
|
||||||
...p,
|
|
||||||
}));
|
|
||||||
setResults(placeResults);
|
|
||||||
setOpen(placeResults.length > 0);
|
|
||||||
} catch (err) {
|
|
||||||
logNonAbortError('places search', err);
|
|
||||||
}
|
|
||||||
}, 200);
|
|
||||||
}, []);
|
|
||||||
|
|
||||||
const handleKeyDown = useCallback(
|
|
||||||
(e: React.KeyboardEvent) => {
|
|
||||||
if (e.key === 'ArrowDown') {
|
|
||||||
e.preventDefault();
|
|
||||||
setActiveIndex((prev) => (prev < results.length - 1 ? prev + 1 : prev));
|
|
||||||
} else if (e.key === 'ArrowUp') {
|
|
||||||
e.preventDefault();
|
|
||||||
setActiveIndex((prev) => (prev > 0 ? prev - 1 : -1));
|
|
||||||
} else if (e.key === 'Enter') {
|
|
||||||
e.preventDefault();
|
|
||||||
if (activeIndex >= 0 && activeIndex < results.length) {
|
|
||||||
selectResult(results[activeIndex]);
|
|
||||||
} else if (looksLikePostcode(query)) {
|
|
||||||
selectPostcode(query);
|
|
||||||
}
|
|
||||||
} else if (e.key === 'Escape') {
|
|
||||||
setOpen(false);
|
|
||||||
inputRef.current?.blur();
|
|
||||||
}
|
|
||||||
},
|
|
||||||
[results, activeIndex, query, selectResult, selectPostcode]
|
|
||||||
);
|
|
||||||
|
|
||||||
// Cleanup on unmount
|
|
||||||
useEffect(() => {
|
|
||||||
return () => {
|
|
||||||
abortRef.current?.abort();
|
|
||||||
if (debounceRef.current) clearTimeout(debounceRef.current);
|
|
||||||
};
|
|
||||||
}, []);
|
|
||||||
|
|
||||||
// Mobile collapsed state: just a search icon button
|
|
||||||
if (isMobile && !expanded) {
|
|
||||||
return (
|
|
||||||
<button
|
|
||||||
type="button"
|
|
||||||
onClick={() => setExpanded(true)}
|
|
||||||
className="absolute top-3 left-3 z-10 p-2 bg-white dark:bg-warm-800 rounded shadow-lg"
|
|
||||||
aria-label="Search places or postcodes"
|
|
||||||
>
|
|
||||||
<SearchIcon className="w-5 h-5 text-warm-600 dark:text-warm-300" />
|
|
||||||
</button>
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
||||||
return (
|
|
||||||
<div ref={containerRef} className="absolute top-3 left-3 z-10 flex flex-col">
|
|
||||||
<div className="relative">
|
|
||||||
<div className="flex items-center shadow-lg rounded overflow-hidden bg-white dark:bg-warm-800">
|
|
||||||
<SearchIcon className="w-4 h-4 text-warm-400 dark:text-warm-500 ml-3 shrink-0" />
|
|
||||||
<input
|
|
||||||
ref={inputRef}
|
|
||||||
type="text"
|
|
||||||
value={query}
|
|
||||||
onChange={(e) => handleInputChange(e.target.value)}
|
|
||||||
onFocus={() => {
|
|
||||||
if (results.length > 0) setOpen(true);
|
|
||||||
}}
|
|
||||||
onKeyDown={handleKeyDown}
|
|
||||||
placeholder="Search places or postcodes..."
|
|
||||||
className="px-2 py-2 text-sm w-56 border-none outline-none bg-transparent text-warm-700 dark:text-warm-200 placeholder-warm-400 dark:placeholder-warm-500"
|
|
||||||
/>
|
|
||||||
{loading && (
|
|
||||||
<div className="mr-3 w-4 h-4 border-2 border-warm-300 dark:border-warm-600 border-t-teal-500 rounded-full animate-spin" />
|
|
||||||
)}
|
|
||||||
</div>
|
|
||||||
|
|
||||||
{open && results.length > 0 && (
|
|
||||||
<div className="absolute top-full left-0 right-0 mt-1 bg-white dark:bg-warm-800 rounded shadow-lg border border-warm-200 dark:border-warm-700 max-h-64 overflow-y-auto">
|
|
||||||
{results.map((result, idx) => (
|
|
||||||
<button
|
|
||||||
key={
|
|
||||||
result.type === 'postcode'
|
|
||||||
? `pc-${result.label}`
|
|
||||||
: `pl-${result.name}-${result.lat}`
|
|
||||||
}
|
|
||||||
type="button"
|
|
||||||
className={`w-full text-left px-3 py-2 flex items-center gap-2 text-sm cursor-pointer ${
|
|
||||||
idx === activeIndex
|
|
||||||
? 'bg-teal-50 dark:bg-teal-900/30'
|
|
||||||
: 'hover:bg-warm-50 dark:hover:bg-warm-700'
|
|
||||||
}`}
|
|
||||||
onMouseEnter={() => setActiveIndex(idx)}
|
|
||||||
onMouseDown={(e) => {
|
|
||||||
e.preventDefault();
|
|
||||||
selectResult(result);
|
|
||||||
}}
|
|
||||||
>
|
|
||||||
{result.type === 'postcode' ? (
|
|
||||||
<>
|
|
||||||
<SearchIcon className="w-4 h-4 text-warm-400 dark:text-warm-500 shrink-0" />
|
|
||||||
<span className="text-warm-700 dark:text-warm-200">{result.label}</span>
|
|
||||||
<span className="text-warm-400 dark:text-warm-500 text-xs ml-auto">
|
|
||||||
postcode
|
|
||||||
</span>
|
|
||||||
</>
|
|
||||||
) : (
|
|
||||||
<>
|
|
||||||
<MapPinIcon className="w-4 h-4 text-warm-400 dark:text-warm-500 shrink-0" />
|
|
||||||
<span className="text-warm-700 dark:text-warm-200">{result.name}</span>
|
|
||||||
<span className="text-warm-400 dark:text-warm-500 text-xs ml-auto">
|
|
||||||
{result.place_type}
|
|
||||||
</span>
|
|
||||||
</>
|
|
||||||
)}
|
|
||||||
</button>
|
|
||||||
))}
|
|
||||||
</div>
|
|
||||||
)}
|
|
||||||
</div>
|
|
||||||
|
|
||||||
{error && (
|
|
||||||
<span className="text-xs text-red-600 dark:text-red-300 bg-white/90 dark:bg-warm-800/90 rounded px-2 py-0.5 shadow mt-1">
|
|
||||||
{error}
|
|
||||||
</span>
|
|
||||||
)}
|
|
||||||
</div>
|
|
||||||
);
|
|
||||||
}
|
|
||||||
|
|
@ -221,7 +221,7 @@ function PropertyCard({ property }: { property: Property }) {
|
||||||
{age !== undefined && (
|
{age !== undefined && (
|
||||||
<div>
|
<div>
|
||||||
<span className="text-warm-500 dark:text-warm-400">Built:</span>{' '}
|
<span className="text-warm-500 dark:text-warm-400">Built:</span>{' '}
|
||||||
{formatAge(age, property.is_construction_date_approximate ?? true)}
|
{formatAge(age, property.is_construction_date_approximate)}
|
||||||
</div>
|
</div>
|
||||||
)}
|
)}
|
||||||
{property.current_energy_rating && (
|
{property.current_energy_rating && (
|
||||||
|
|
|
||||||
|
|
@ -1,10 +1,52 @@
|
||||||
|
import { useEffect, useState } from 'react';
|
||||||
import type { HexagonLocation } from '../../lib/external-search';
|
import type { HexagonLocation } from '../../lib/external-search';
|
||||||
|
import { apiUrl, logNonAbortError } from '../../lib/api';
|
||||||
|
|
||||||
interface StreetViewEmbedProps {
|
interface StreetViewEmbedProps {
|
||||||
location: HexagonLocation;
|
location: HexagonLocation;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type Status = 'loading' | 'ok' | 'none' | 'error';
|
||||||
|
|
||||||
export default function StreetViewEmbed({ location }: StreetViewEmbedProps) {
|
export default function StreetViewEmbed({ location }: StreetViewEmbedProps) {
|
||||||
|
const [status, setStatus] = useState<Status>('loading');
|
||||||
|
const [panoId, setPanoId] = useState<string | null>(null);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
setStatus('loading');
|
||||||
|
setPanoId(null);
|
||||||
|
|
||||||
|
const controller = new AbortController();
|
||||||
|
const params = new URLSearchParams({
|
||||||
|
lat: String(location.lat),
|
||||||
|
lon: String(location.lon),
|
||||||
|
});
|
||||||
|
|
||||||
|
fetch(apiUrl('streetview', params), { signal: controller.signal })
|
||||||
|
.then((res) => {
|
||||||
|
if (!res.ok) throw new Error(`HTTP ${res.status}`);
|
||||||
|
return res.json();
|
||||||
|
})
|
||||||
|
.then((data: { status: string; pano_id?: string }) => {
|
||||||
|
if (data.status === 'OK' && data.pano_id) {
|
||||||
|
setPanoId(data.pano_id);
|
||||||
|
setStatus('ok');
|
||||||
|
} else {
|
||||||
|
setStatus('none');
|
||||||
|
}
|
||||||
|
})
|
||||||
|
.catch((err) => {
|
||||||
|
logNonAbortError('streetview', err);
|
||||||
|
if (!controller.signal.aborted) {
|
||||||
|
setStatus('error');
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
return () => controller.abort();
|
||||||
|
}, [location.lat, location.lon]);
|
||||||
|
|
||||||
|
if (status === 'none' || status === 'error') return null;
|
||||||
|
|
||||||
return (
|
return (
|
||||||
<div>
|
<div>
|
||||||
<div className="px-3 py-1.5 text-xs font-bold text-warm-500 bg-warm-50 dark:bg-warm-900 dark:text-warm-400 sticky top-0">
|
<div className="px-3 py-1.5 text-xs font-bold text-warm-500 bg-warm-50 dark:bg-warm-900 dark:text-warm-400 sticky top-0">
|
||||||
|
|
@ -12,13 +54,20 @@ export default function StreetViewEmbed({ location }: StreetViewEmbedProps) {
|
||||||
</div>
|
</div>
|
||||||
<div className="px-3 py-2">
|
<div className="px-3 py-2">
|
||||||
<div className="rounded overflow-hidden border border-warm-200 dark:border-warm-700">
|
<div className="rounded overflow-hidden border border-warm-200 dark:border-warm-700">
|
||||||
<iframe
|
{status === 'loading' ? (
|
||||||
className="w-full"
|
<div
|
||||||
style={{ height: 240, border: 0 }}
|
className="w-full animate-pulse bg-warm-200 dark:bg-warm-700"
|
||||||
loading="lazy"
|
style={{ height: 240 }}
|
||||||
referrerPolicy="no-referrer-when-downgrade"
|
/>
|
||||||
src={`https://maps.google.com/maps?layer=c&cbll=${location.lat},${location.lon}&cbp=11,0,0,0,0&output=svembed`}
|
) : (
|
||||||
/>
|
<iframe
|
||||||
|
className="w-full"
|
||||||
|
style={{ height: 240, border: 0 }}
|
||||||
|
loading="lazy"
|
||||||
|
referrerPolicy="no-referrer-when-downgrade"
|
||||||
|
src={`https://maps.google.com/maps?layer=c&panoid=${panoId}&cbp=11,0,0,0,0&output=svembed`}
|
||||||
|
/>
|
||||||
|
)}
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
|
||||||
|
|
@ -1,61 +1,69 @@
|
||||||
import { useState, useCallback } from 'react';
|
import { useState, useCallback, useRef, useEffect } from 'react';
|
||||||
import { Slider } from '../ui/Slider';
|
import { Slider } from '../ui/Slider';
|
||||||
import { PillToggle } from '../ui/PillToggle';
|
|
||||||
import { PillGroup } from '../ui/PillGroup';
|
|
||||||
import { IconButton } from '../ui/IconButton';
|
import { IconButton } from '../ui/IconButton';
|
||||||
|
import { PlaceSearchInput } from '../ui/PlaceSearchInput';
|
||||||
import { CloseIcon } from '../ui/icons/CloseIcon';
|
import { CloseIcon } from '../ui/icons/CloseIcon';
|
||||||
import { MapPinIcon } from '../ui/icons/MapPinIcon';
|
import { MapPinIcon } from '../ui/icons/MapPinIcon';
|
||||||
import { RouteIcon } from '../ui/icons/RouteIcon';
|
import { RouteIcon } from '../ui/icons/RouteIcon';
|
||||||
import { formatFilterValue } from '../../lib/format';
|
import { formatFilterValue } from '../../lib/format';
|
||||||
import { authHeaders } from '../../lib/api';
|
import { authHeaders, logNonAbortError } from '../../lib/api';
|
||||||
import type { TransportMode } from '../../hooks/useTravelTime';
|
import { useLocationSearch, type SearchResult } from '../../hooks/useLocationSearch';
|
||||||
|
import { MODE_LABELS, type TransportMode } from '../../hooks/useTravelTime';
|
||||||
const MODES: { value: TransportMode; label: string }[] = [
|
|
||||||
{ value: 'car', label: 'Car' },
|
|
||||||
{ value: 'bicycle', label: 'Bicycle' },
|
|
||||||
{ value: 'walking', label: 'Walking' },
|
|
||||||
{ value: 'transit', label: 'Transit' },
|
|
||||||
];
|
|
||||||
|
|
||||||
interface TravelTimeCardProps {
|
interface TravelTimeCardProps {
|
||||||
|
mode: TransportMode;
|
||||||
destination: [number, number] | null;
|
destination: [number, number] | null;
|
||||||
destinationLabel: string;
|
destinationLabel: string;
|
||||||
mode: TransportMode;
|
|
||||||
timeRange: [number, number] | null;
|
timeRange: [number, number] | null;
|
||||||
dataRange: [number, number] | null;
|
dataRange: [number, number] | null;
|
||||||
onSetDestination: (lat: number, lon: number, label: string) => void;
|
onSetDestination: (lat: number, lon: number, label: string) => void;
|
||||||
onModeChange: (mode: TransportMode) => void;
|
|
||||||
onTimeRangeChange: (range: [number, number]) => void;
|
onTimeRangeChange: (range: [number, number]) => void;
|
||||||
onRemove: () => void;
|
onRemove: () => void;
|
||||||
}
|
}
|
||||||
|
|
||||||
export function TravelTimeCard({
|
export function TravelTimeCard({
|
||||||
|
mode,
|
||||||
destination,
|
destination,
|
||||||
destinationLabel,
|
destinationLabel,
|
||||||
mode,
|
|
||||||
timeRange,
|
timeRange,
|
||||||
dataRange,
|
dataRange,
|
||||||
onSetDestination,
|
onSetDestination,
|
||||||
onModeChange,
|
|
||||||
onTimeRangeChange,
|
onTimeRangeChange,
|
||||||
onRemove,
|
onRemove,
|
||||||
}: TravelTimeCardProps) {
|
}: TravelTimeCardProps) {
|
||||||
const [query, setQuery] = useState('');
|
const search = useLocationSearch();
|
||||||
const [loading, setLoading] = useState(false);
|
const [loading, setLoading] = useState(false);
|
||||||
const [error, setError] = useState<string | null>(null);
|
const [error, setError] = useState<string | null>(null);
|
||||||
|
const containerRef = useRef<HTMLDivElement>(null);
|
||||||
|
|
||||||
const handleSearch = useCallback(
|
// Close dropdown on outside click
|
||||||
async (e: React.FormEvent) => {
|
useEffect(() => {
|
||||||
e.preventDefault();
|
const handler = (e: MouseEvent) => {
|
||||||
const trimmed = query.trim();
|
if (containerRef.current && !containerRef.current.contains(e.target as Node)) {
|
||||||
if (!trimmed) return;
|
search.close();
|
||||||
|
}
|
||||||
|
};
|
||||||
|
document.addEventListener('mousedown', handler);
|
||||||
|
return () => document.removeEventListener('mousedown', handler);
|
||||||
|
}, [search]);
|
||||||
|
|
||||||
|
const selectResult = useCallback(
|
||||||
|
async (result: SearchResult) => {
|
||||||
|
if (result.type === 'place') {
|
||||||
|
onSetDestination(result.lat, result.lon, result.name);
|
||||||
|
search.clear();
|
||||||
|
setError(null);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Postcode — fetch coordinates
|
||||||
setError(null);
|
setError(null);
|
||||||
setLoading(true);
|
setLoading(true);
|
||||||
|
search.close();
|
||||||
try {
|
try {
|
||||||
const res = await fetch(
|
const res = await fetch(
|
||||||
`/api/postcode/${encodeURIComponent(trimmed)}`,
|
`/api/postcode/${encodeURIComponent(result.label)}`,
|
||||||
authHeaders()
|
authHeaders(),
|
||||||
);
|
);
|
||||||
if (!res.ok) {
|
if (!res.ok) {
|
||||||
setError('Postcode not found');
|
setError('Postcode not found');
|
||||||
|
|
@ -64,14 +72,15 @@ export function TravelTimeCard({
|
||||||
const json: { postcode: string; latitude: number; longitude: number } =
|
const json: { postcode: string; latitude: number; longitude: number } =
|
||||||
await res.json();
|
await res.json();
|
||||||
onSetDestination(json.latitude, json.longitude, json.postcode);
|
onSetDestination(json.latitude, json.longitude, json.postcode);
|
||||||
setQuery('');
|
search.clear();
|
||||||
} catch {
|
} catch (err) {
|
||||||
|
logNonAbortError('Postcode lookup failed', err);
|
||||||
setError('Lookup failed');
|
setError('Lookup failed');
|
||||||
} finally {
|
} finally {
|
||||||
setLoading(false);
|
setLoading(false);
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
[query, onSetDestination]
|
[onSetDestination, search],
|
||||||
);
|
);
|
||||||
|
|
||||||
const sliderMin = dataRange ? Math.floor(dataRange[0]) : 0;
|
const sliderMin = dataRange ? Math.floor(dataRange[0]) : 0;
|
||||||
|
|
@ -85,7 +94,7 @@ export function TravelTimeCard({
|
||||||
<div className="flex items-center gap-1.5">
|
<div className="flex items-center gap-1.5">
|
||||||
<RouteIcon className="w-4 h-4 text-teal-600 dark:text-teal-400" />
|
<RouteIcon className="w-4 h-4 text-teal-600 dark:text-teal-400" />
|
||||||
<span className="text-sm font-medium text-navy-950 dark:text-warm-100">
|
<span className="text-sm font-medium text-navy-950 dark:text-warm-100">
|
||||||
Travel Time
|
Travel Time ({MODE_LABELS[mode]})
|
||||||
</span>
|
</span>
|
||||||
</div>
|
</div>
|
||||||
<IconButton onClick={() => onRemove()} title="Remove travel time">
|
<IconButton onClick={() => onRemove()} title="Remove travel time">
|
||||||
|
|
@ -94,26 +103,17 @@ export function TravelTimeCard({
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
{/* Destination search */}
|
{/* Destination search */}
|
||||||
<div>
|
<div ref={containerRef} className="relative">
|
||||||
<form onSubmit={handleSearch} className="flex gap-1">
|
<PlaceSearchInput
|
||||||
<input
|
search={search}
|
||||||
type="text"
|
onSelect={selectResult}
|
||||||
value={query}
|
loading={loading}
|
||||||
onChange={(e) => {
|
placeholder={destination ? 'Change destination...' : 'Search destination...'}
|
||||||
setQuery(e.target.value);
|
size="xs"
|
||||||
setError(null);
|
inputClassName="w-full px-2 py-1 text-xs rounded border border-warm-200 dark:border-warm-600 bg-white dark:bg-warm-800 text-navy-950 dark:text-warm-200 placeholder-warm-400 dark:placeholder-warm-500 outline-none focus:ring-1 focus:ring-teal-400"
|
||||||
}}
|
onInputChange={() => setError(null)}
|
||||||
placeholder={destination ? 'Change destination...' : 'Enter postcode...'}
|
/>
|
||||||
className="flex-1 min-w-0 px-2 py-1 text-xs rounded border border-warm-200 dark:border-warm-600 bg-white dark:bg-warm-800 text-navy-950 dark:text-warm-200 placeholder-warm-400 dark:placeholder-warm-500 outline-none focus:ring-1 focus:ring-teal-400"
|
|
||||||
/>
|
|
||||||
<button
|
|
||||||
type="submit"
|
|
||||||
disabled={loading || !query.trim()}
|
|
||||||
className="px-2 py-1 text-xs rounded bg-teal-600 text-white hover:bg-teal-700 disabled:opacity-50"
|
|
||||||
>
|
|
||||||
{loading ? '...' : 'Go'}
|
|
||||||
</button>
|
|
||||||
</form>
|
|
||||||
{error && (
|
{error && (
|
||||||
<p className="text-xs text-red-600 dark:text-red-400 mt-0.5">{error}</p>
|
<p className="text-xs text-red-600 dark:text-red-400 mt-0.5">{error}</p>
|
||||||
)}
|
)}
|
||||||
|
|
@ -127,24 +127,6 @@ export function TravelTimeCard({
|
||||||
)}
|
)}
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
{/* Mode selector */}
|
|
||||||
<div>
|
|
||||||
<span className="text-[10px] font-medium text-warm-500 dark:text-warm-400 uppercase tracking-wide">
|
|
||||||
Mode
|
|
||||||
</span>
|
|
||||||
<PillGroup className="mt-0.5">
|
|
||||||
{MODES.map((m) => (
|
|
||||||
<PillToggle
|
|
||||||
key={m.value}
|
|
||||||
label={m.label}
|
|
||||||
active={mode === m.value}
|
|
||||||
onClick={() => onModeChange(m.value)}
|
|
||||||
size="xs"
|
|
||||||
/>
|
|
||||||
))}
|
|
||||||
</PillGroup>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
{/* Time range slider — only show when we have data */}
|
{/* Time range slider — only show when we have data */}
|
||||||
{destination && dataRange && (
|
{destination && dataRange && (
|
||||||
<div>
|
<div>
|
||||||
|
|
|
||||||
123
frontend/src/components/ui/PlaceSearchInput.tsx
Normal file
123
frontend/src/components/ui/PlaceSearchInput.tsx
Normal file
|
|
@ -0,0 +1,123 @@
|
||||||
|
import type React from 'react';
|
||||||
|
import type { SearchResult } from '../../hooks/useLocationSearch';
|
||||||
|
import { SearchIcon } from './icons/SearchIcon';
|
||||||
|
import { MapPinIcon } from './icons/MapPinIcon';
|
||||||
|
|
||||||
|
interface SearchHook {
|
||||||
|
query: string;
|
||||||
|
results: SearchResult[];
|
||||||
|
activeIndex: number;
|
||||||
|
setActiveIndex: (idx: number) => void;
|
||||||
|
open: boolean;
|
||||||
|
setOpen: (open: boolean) => void;
|
||||||
|
handleInputChange: (value: string) => void;
|
||||||
|
handleKeyDown: (
|
||||||
|
e: React.KeyboardEvent,
|
||||||
|
onSelect: (result: SearchResult) => void,
|
||||||
|
) => void;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface PlaceSearchInputProps {
|
||||||
|
search: SearchHook;
|
||||||
|
onSelect: (result: SearchResult) => void;
|
||||||
|
loading?: boolean;
|
||||||
|
placeholder?: string;
|
||||||
|
size?: 'sm' | 'xs';
|
||||||
|
inputClassName?: string;
|
||||||
|
inputRef?: React.Ref<HTMLInputElement>;
|
||||||
|
onInputChange?: () => void;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function PlaceSearchInput({
|
||||||
|
search,
|
||||||
|
onSelect,
|
||||||
|
loading,
|
||||||
|
placeholder,
|
||||||
|
size = 'sm',
|
||||||
|
inputClassName,
|
||||||
|
inputRef,
|
||||||
|
onInputChange,
|
||||||
|
}: PlaceSearchInputProps) {
|
||||||
|
const sm = size === 'sm';
|
||||||
|
const iconSize = sm ? 'w-4 h-4' : 'w-3 h-3';
|
||||||
|
const spinnerSize = sm ? 'w-4 h-4' : 'w-3 h-3';
|
||||||
|
|
||||||
|
return (
|
||||||
|
<div className="relative flex-1 min-w-0">
|
||||||
|
<input
|
||||||
|
ref={inputRef}
|
||||||
|
type="text"
|
||||||
|
value={search.query}
|
||||||
|
onChange={(e) => {
|
||||||
|
search.handleInputChange(e.target.value);
|
||||||
|
onInputChange?.();
|
||||||
|
}}
|
||||||
|
onFocus={() => {
|
||||||
|
if (search.results.length > 0) search.setOpen(true);
|
||||||
|
}}
|
||||||
|
onKeyDown={(e) => search.handleKeyDown(e, onSelect)}
|
||||||
|
placeholder={placeholder}
|
||||||
|
className={inputClassName}
|
||||||
|
/>
|
||||||
|
|
||||||
|
{loading && (
|
||||||
|
<div
|
||||||
|
className={`absolute right-2 top-1/2 -translate-y-1/2 ${spinnerSize} border-2 border-warm-300 dark:border-warm-600 border-t-teal-500 rounded-full animate-spin`}
|
||||||
|
/>
|
||||||
|
)}
|
||||||
|
|
||||||
|
{search.open && search.results.length > 0 && (
|
||||||
|
<div
|
||||||
|
className={`absolute top-full left-0 right-0 mt-1 bg-white dark:bg-warm-800 rounded shadow-lg border border-warm-200 dark:border-warm-700 ${sm ? 'max-h-64' : 'max-h-48'} overflow-y-auto z-20`}
|
||||||
|
>
|
||||||
|
{search.results.map((result, idx) => (
|
||||||
|
<button
|
||||||
|
key={
|
||||||
|
result.type === 'postcode'
|
||||||
|
? `pc-${result.label}`
|
||||||
|
: `pl-${result.name}-${result.lat}`
|
||||||
|
}
|
||||||
|
type="button"
|
||||||
|
className={`w-full text-left flex items-center cursor-pointer ${
|
||||||
|
sm ? 'px-3 py-2 gap-2 text-sm' : 'px-2 py-1.5 gap-1.5 text-xs'
|
||||||
|
} ${
|
||||||
|
idx === search.activeIndex
|
||||||
|
? 'bg-teal-50 dark:bg-teal-900/30'
|
||||||
|
: 'hover:bg-warm-50 dark:hover:bg-warm-700'
|
||||||
|
}`}
|
||||||
|
onMouseEnter={() => search.setActiveIndex(idx)}
|
||||||
|
onMouseDown={(e) => {
|
||||||
|
e.preventDefault();
|
||||||
|
onSelect(result);
|
||||||
|
}}
|
||||||
|
>
|
||||||
|
{result.type === 'postcode' ? (
|
||||||
|
<>
|
||||||
|
<SearchIcon
|
||||||
|
className={`${iconSize} text-warm-400 dark:text-warm-500 shrink-0`}
|
||||||
|
/>
|
||||||
|
<span className="text-warm-700 dark:text-warm-200">{result.label}</span>
|
||||||
|
</>
|
||||||
|
) : (
|
||||||
|
<>
|
||||||
|
<MapPinIcon
|
||||||
|
className={`${iconSize} text-warm-400 dark:text-warm-500 shrink-0`}
|
||||||
|
/>
|
||||||
|
<span className="text-warm-700 dark:text-warm-200">
|
||||||
|
{result.name}
|
||||||
|
{result.city && (
|
||||||
|
<span className="text-warm-400 dark:text-warm-500">
|
||||||
|
{' '}
|
||||||
|
({result.city})
|
||||||
|
</span>
|
||||||
|
)}
|
||||||
|
</span>
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
</button>
|
||||||
|
))}
|
||||||
|
</div>
|
||||||
|
)}
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
@ -2,24 +2,32 @@ import { useState, useCallback, useRef } from 'react';
|
||||||
import type { FeatureFilters } from '../types';
|
import type { FeatureFilters } from '../types';
|
||||||
import { apiUrl, authHeaders, logNonAbortError } from '../lib/api';
|
import { apiUrl, authHeaders, logNonAbortError } from '../lib/api';
|
||||||
|
|
||||||
|
interface AiFiltersResult {
|
||||||
|
filters: FeatureFilters;
|
||||||
|
notes: string;
|
||||||
|
}
|
||||||
|
|
||||||
interface UseAiFiltersResult {
|
interface UseAiFiltersResult {
|
||||||
fetchAiFilters: (query: string) => Promise<FeatureFilters | null>;
|
fetchAiFilters: (query: string) => Promise<AiFiltersResult | null>;
|
||||||
loading: boolean;
|
loading: boolean;
|
||||||
error: string | null;
|
error: string | null;
|
||||||
|
notes: string | null;
|
||||||
}
|
}
|
||||||
|
|
||||||
export function useAiFilters(): UseAiFiltersResult {
|
export function useAiFilters(): UseAiFiltersResult {
|
||||||
const [loading, setLoading] = useState(false);
|
const [loading, setLoading] = useState(false);
|
||||||
const [error, setError] = useState<string | null>(null);
|
const [error, setError] = useState<string | null>(null);
|
||||||
|
const [notes, setNotes] = useState<string | null>(null);
|
||||||
const abortRef = useRef<AbortController | null>(null);
|
const abortRef = useRef<AbortController | null>(null);
|
||||||
|
|
||||||
const fetchAiFilters = useCallback(async (query: string): Promise<FeatureFilters | null> => {
|
const fetchAiFilters = useCallback(async (query: string): Promise<AiFiltersResult | null> => {
|
||||||
abortRef.current?.abort();
|
abortRef.current?.abort();
|
||||||
const controller = new AbortController();
|
const controller = new AbortController();
|
||||||
abortRef.current = controller;
|
abortRef.current = controller;
|
||||||
|
|
||||||
setLoading(true);
|
setLoading(true);
|
||||||
setError(null);
|
setError(null);
|
||||||
|
setNotes(null);
|
||||||
|
|
||||||
try {
|
try {
|
||||||
const url = apiUrl('ai-filters');
|
const url = apiUrl('ai-filters');
|
||||||
|
|
@ -39,8 +47,13 @@ export function useAiFilters(): UseAiFiltersResult {
|
||||||
}
|
}
|
||||||
|
|
||||||
const json = await response.json();
|
const json = await response.json();
|
||||||
|
const result: AiFiltersResult = {
|
||||||
|
filters: json.filters as FeatureFilters,
|
||||||
|
notes: json.notes || '',
|
||||||
|
};
|
||||||
|
setNotes(result.notes || null);
|
||||||
setLoading(false);
|
setLoading(false);
|
||||||
return json.filters as FeatureFilters;
|
return result;
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
if (controller.signal.aborted) return null;
|
if (controller.signal.aborted) return null;
|
||||||
logNonAbortError('ai-filters', err);
|
logNonAbortError('ai-filters', err);
|
||||||
|
|
@ -51,5 +64,5 @@ export function useAiFilters(): UseAiFiltersResult {
|
||||||
}
|
}
|
||||||
}, []);
|
}, []);
|
||||||
|
|
||||||
return { fetchAiFilters, loading, error };
|
return { fetchAiFilters, loading, error, notes };
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -7,13 +7,14 @@ export interface AuthUser {
|
||||||
verified: boolean;
|
verified: boolean;
|
||||||
}
|
}
|
||||||
|
|
||||||
// PocketBase RecordModel stores user fields as dynamic properties
|
function recordToUser(record: { id: string; [key: string]: unknown }): AuthUser {
|
||||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
if (typeof record.email !== 'string') {
|
||||||
function recordToUser(record: any): AuthUser {
|
throw new Error('PocketBase record missing email field');
|
||||||
|
}
|
||||||
return {
|
return {
|
||||||
id: record.id || '',
|
id: record.id,
|
||||||
email: record.email || '',
|
email: record.email,
|
||||||
verified: record.verified || false,
|
verified: typeof record.verified === 'boolean' ? record.verified : false,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -6,13 +6,18 @@ import type {
|
||||||
HexagonData,
|
HexagonData,
|
||||||
PostcodeFeature,
|
PostcodeFeature,
|
||||||
PostcodeProperties,
|
PostcodeProperties,
|
||||||
|
PostcodeGeometry,
|
||||||
POI,
|
POI,
|
||||||
FeatureMeta,
|
FeatureMeta,
|
||||||
Bounds,
|
Bounds,
|
||||||
} from '../types';
|
} from '../types';
|
||||||
import type { SearchedPostcode } from '../components/map/PostcodeSearch';
|
|
||||||
import { DENSITY_GRADIENT, DENSITY_GRADIENT_DARK } from '../lib/consts';
|
import { DENSITY_GRADIENT, DENSITY_GRADIENT_DARK } from '../lib/consts';
|
||||||
import { emojiToTwemojiUrl, getFeatureFillColor } from '../lib/map-utils';
|
import { emojiToTwemojiUrl, getFeatureFillColor } from '../lib/map-utils';
|
||||||
|
import {
|
||||||
|
TRANSPORT_MODES,
|
||||||
|
type TransportMode,
|
||||||
|
type TravelTimeEntries,
|
||||||
|
} from './useTravelTime';
|
||||||
|
|
||||||
/** Convert POI id (e.g. "n12345") to OpenStreetMap URL */
|
/** Convert POI id (e.g. "n12345") to OpenStreetMap URL */
|
||||||
function osmIdToUrl(id: string): string | null {
|
function osmIdToUrl(id: string): string | null {
|
||||||
|
|
@ -38,12 +43,10 @@ interface UseDeckLayersProps {
|
||||||
onHexagonClick: (id: string, isPostcode?: boolean) => void;
|
onHexagonClick: (id: string, isPostcode?: boolean) => void;
|
||||||
onHexagonHover: (h3: string | null, x?: number, y?: number) => void;
|
onHexagonHover: (h3: string | null, x?: number, y?: number) => void;
|
||||||
theme: 'light' | 'dark';
|
theme: 'light' | 'dark';
|
||||||
searchedPostcode?: SearchedPostcode | null;
|
selectedPostcodeGeometry?: PostcodeGeometry | null;
|
||||||
bounds?: Bounds | null;
|
bounds?: Bounds | null;
|
||||||
travelTimeEnabled?: boolean;
|
travelTimeEntries?: TravelTimeEntries;
|
||||||
travelTimeDestination?: [number, number] | null;
|
travelTimeColorRanges?: Partial<Record<TransportMode, [number, number]>>;
|
||||||
travelTimeColorRange?: [number, number] | null;
|
|
||||||
travelTimeRange?: [number, number] | null;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface PopupInfo {
|
export interface PopupInfo {
|
||||||
|
|
@ -54,6 +57,17 @@ export interface PopupInfo {
|
||||||
id: string;
|
id: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/** Find the primary travel mode: first mode (in canonical order) with a destination and color range. */
|
||||||
|
function getPrimaryTravelMode(
|
||||||
|
entries: TravelTimeEntries,
|
||||||
|
colorRanges: Partial<Record<TransportMode, [number, number]>>
|
||||||
|
): TransportMode | null {
|
||||||
|
for (const mode of TRANSPORT_MODES) {
|
||||||
|
if (entries[mode]?.destination && colorRanges[mode]) return mode;
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
export function useDeckLayers({
|
export function useDeckLayers({
|
||||||
data,
|
data,
|
||||||
postcodeData,
|
postcodeData,
|
||||||
|
|
@ -68,12 +82,10 @@ export function useDeckLayers({
|
||||||
onHexagonClick,
|
onHexagonClick,
|
||||||
onHexagonHover,
|
onHexagonHover,
|
||||||
theme,
|
theme,
|
||||||
searchedPostcode,
|
selectedPostcodeGeometry,
|
||||||
bounds: viewportBounds,
|
bounds: viewportBounds,
|
||||||
travelTimeEnabled = false,
|
travelTimeEntries = {},
|
||||||
travelTimeDestination,
|
travelTimeColorRanges = {},
|
||||||
travelTimeColorRange,
|
|
||||||
travelTimeRange,
|
|
||||||
}: UseDeckLayersProps) {
|
}: UseDeckLayersProps) {
|
||||||
const [popupInfo, setPopupInfo] = useState<PopupInfo | null>(null);
|
const [popupInfo, setPopupInfo] = useState<PopupInfo | null>(null);
|
||||||
const [hoverPosition, setHoverPosition] = useState<{ x: number; y: number } | null>(null);
|
const [hoverPosition, setHoverPosition] = useState<{ x: number; y: number } | null>(null);
|
||||||
|
|
@ -103,14 +115,17 @@ export function useDeckLayers({
|
||||||
const hoveredPostcodeRef = useRef(hoveredPostcode);
|
const hoveredPostcodeRef = useRef(hoveredPostcode);
|
||||||
hoveredPostcodeRef.current = hoveredPostcode;
|
hoveredPostcodeRef.current = hoveredPostcode;
|
||||||
|
|
||||||
const travelTimeEnabledRef = useRef(travelTimeEnabled);
|
const travelTimeEntriesRef = useRef(travelTimeEntries);
|
||||||
travelTimeEnabledRef.current = travelTimeEnabled;
|
travelTimeEntriesRef.current = travelTimeEntries;
|
||||||
const travelTimeDestinationRef = useRef(travelTimeDestination);
|
const travelTimeColorRangesRef = useRef(travelTimeColorRanges);
|
||||||
travelTimeDestinationRef.current = travelTimeDestination;
|
travelTimeColorRangesRef.current = travelTimeColorRanges;
|
||||||
const travelTimeColorRangeRef = useRef(travelTimeColorRange);
|
|
||||||
travelTimeColorRangeRef.current = travelTimeColorRange;
|
const primaryTravelMode = useMemo(
|
||||||
const travelTimeRangeRef = useRef(travelTimeRange);
|
() => getPrimaryTravelMode(travelTimeEntries, travelTimeColorRanges),
|
||||||
travelTimeRangeRef.current = travelTimeRange;
|
[travelTimeEntries, travelTimeColorRanges]
|
||||||
|
);
|
||||||
|
const primaryTravelModeRef = useRef(primaryTravelMode);
|
||||||
|
primaryTravelModeRef.current = primaryTravelMode;
|
||||||
|
|
||||||
const colorFeatureMeta = useMemo(
|
const colorFeatureMeta = useMemo(
|
||||||
() => (viewFeature ? features.find((f) => f.name === viewFeature) || null : null),
|
() => (viewFeature ? features.find((f) => f.name === viewFeature) || null : null),
|
||||||
|
|
@ -238,7 +253,17 @@ export function useDeckLayers({
|
||||||
}, []);
|
}, []);
|
||||||
|
|
||||||
// --- Color triggers ---
|
// --- Color triggers ---
|
||||||
const ttTrigger = `${travelTimeEnabled}|${travelTimeColorRange?.[0]}|${travelTimeColorRange?.[1]}|${travelTimeRange?.[0]}|${travelTimeRange?.[1]}|${travelTimeDestination?.[0]}|${travelTimeDestination?.[1]}`;
|
// Build travel time trigger from all entries
|
||||||
|
const ttTrigger = useMemo(() => {
|
||||||
|
const parts: string[] = [];
|
||||||
|
for (const mode of TRANSPORT_MODES) {
|
||||||
|
const entry = travelTimeEntries[mode];
|
||||||
|
const cr = travelTimeColorRanges[mode];
|
||||||
|
parts.push(`${mode}:${entry?.destination?.[0]}|${entry?.destination?.[1]}|${cr?.[0]}|${cr?.[1]}|${entry?.timeRange?.[0]}|${entry?.timeRange?.[1]}`);
|
||||||
|
}
|
||||||
|
return parts.join(';');
|
||||||
|
}, [travelTimeEntries, travelTimeColorRanges]);
|
||||||
|
|
||||||
const colorTrigger = `${viewFeature}|${colorRange?.[0]}|${colorRange?.[1]}|${filterRange?.[0]}|${filterRange?.[1]}|${countRange.min}|${countRange.max}|${selectedHexagonId}|${hoveredHexagonId}|${theme}|${ttTrigger}`;
|
const colorTrigger = `${viewFeature}|${colorRange?.[0]}|${colorRange?.[1]}|${filterRange?.[0]}|${filterRange?.[1]}|${countRange.min}|${countRange.max}|${selectedHexagonId}|${hoveredHexagonId}|${theme}|${ttTrigger}`;
|
||||||
const postcodeColorTrigger = `${viewFeature}|${colorRange?.[0]}|${colorRange?.[1]}|${filterRange?.[0]}|${filterRange?.[1]}|${postcodeCountRange.min}|${postcodeCountRange.max}|${selectedPostcode}|${hoveredPostcode}|${theme}|${ttTrigger}`;
|
const postcodeColorTrigger = `${viewFeature}|${colorRange?.[0]}|${colorRange?.[1]}|${filterRange?.[0]}|${filterRange?.[1]}|${postcodeCountRange.min}|${postcodeCountRange.max}|${selectedPostcode}|${hoveredPostcode}|${theme}|${ttTrigger}`;
|
||||||
|
|
||||||
|
|
@ -251,17 +276,28 @@ export function useDeckLayers({
|
||||||
getHexagon: (d) => d.h3,
|
getHexagon: (d) => d.h3,
|
||||||
getFillColor: (d) => {
|
getFillColor: (d) => {
|
||||||
const dark = isDarkRef.current;
|
const dark = isDarkRef.current;
|
||||||
// Travel time coloring takes priority
|
const pm = primaryTravelModeRef.current;
|
||||||
if (travelTimeEnabledRef.current && travelTimeDestinationRef.current) {
|
const entries = travelTimeEntriesRef.current;
|
||||||
const ttVal = d.travel_time;
|
const colorRanges = travelTimeColorRangesRef.current;
|
||||||
const ttClr = travelTimeColorRangeRef.current;
|
|
||||||
|
// Travel time coloring: primary mode colors, others dim-filter
|
||||||
|
if (pm) {
|
||||||
|
const ttVal = d[`travel_time_${pm}`];
|
||||||
|
const ttClr = colorRanges[pm];
|
||||||
if (ttVal == null) {
|
if (ttVal == null) {
|
||||||
return (dark ? [80, 70, 65, 80] : [128, 128, 128, 80]) as [number, number, number, number];
|
return (dark ? [80, 70, 65, 80] : [128, 128, 128, 80]) as [number, number, number, number];
|
||||||
}
|
}
|
||||||
const ttFr = travelTimeRangeRef.current;
|
|
||||||
if (ttFr && ((ttVal as number) < ttFr[0] || (ttVal as number) > ttFr[1])) {
|
// Check all modes with time ranges as filters (including primary)
|
||||||
return (dark ? [60, 55, 50, 60] : [180, 180, 180, 60]) as [number, number, number, number];
|
for (const mode of TRANSPORT_MODES) {
|
||||||
|
const entry = entries[mode];
|
||||||
|
if (!entry?.timeRange) continue;
|
||||||
|
const modeVal = d[`travel_time_${mode}`];
|
||||||
|
if (modeVal == null || (modeVal as number) < entry.timeRange[0] || (modeVal as number) > entry.timeRange[1]) {
|
||||||
|
return (dark ? [60, 55, 50, 60] : [180, 180, 180, 60]) as [number, number, number, number];
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if (ttClr) {
|
if (ttClr) {
|
||||||
return getFeatureFillColor(
|
return getFeatureFillColor(
|
||||||
ttVal as number,
|
ttVal as number,
|
||||||
|
|
@ -464,19 +500,19 @@ export function useDeckLayers({
|
||||||
[pois, stablePoiHover]
|
[pois, stablePoiHover]
|
||||||
);
|
);
|
||||||
|
|
||||||
// Check if the searched postcode has data (passes current filters)
|
// Check if the selected postcode has data (passes current filters)
|
||||||
const searchedPostcodeHasData = useMemo(() => {
|
const selectedPostcodeHasData = useMemo(() => {
|
||||||
if (!searchedPostcode) return false;
|
if (!selectedPostcodeGeometry || !selectedHexagonId) return false;
|
||||||
return postcodeData.some((f) => f.properties.postcode === searchedPostcode.postcode);
|
return postcodeData.some((f) => f.properties.postcode === selectedHexagonId);
|
||||||
}, [searchedPostcode, postcodeData]);
|
}, [selectedPostcodeGeometry, selectedHexagonId, postcodeData]);
|
||||||
|
|
||||||
// Highlight layer for searched postcode
|
// Highlight layer for selected postcode (from search)
|
||||||
const searchedPostcodeHighlightLayer = useMemo(() => {
|
const selectedPostcodeHighlightLayer = useMemo(() => {
|
||||||
if (!searchedPostcode) return null;
|
if (!selectedPostcodeGeometry) return null;
|
||||||
const hasData = searchedPostcodeHasData;
|
const hasData = selectedPostcodeHasData;
|
||||||
const feature = {
|
const feature = {
|
||||||
type: 'Feature' as const,
|
type: 'Feature' as const,
|
||||||
geometry: searchedPostcode.geometry,
|
geometry: selectedPostcodeGeometry,
|
||||||
properties: {},
|
properties: {},
|
||||||
};
|
};
|
||||||
return new GeoJsonLayer({
|
return new GeoJsonLayer({
|
||||||
|
|
@ -494,13 +530,25 @@ export function useDeckLayers({
|
||||||
filled: true,
|
filled: true,
|
||||||
pickable: false,
|
pickable: false,
|
||||||
});
|
});
|
||||||
}, [searchedPostcode, searchedPostcodeHasData]);
|
}, [selectedPostcodeGeometry, selectedPostcodeHasData]);
|
||||||
|
|
||||||
|
// Destination markers: one red dot per mode with a destination
|
||||||
|
const destinationMarkerData = useMemo(() => {
|
||||||
|
const points: { position: [number, number] }[] = [];
|
||||||
|
for (const mode of TRANSPORT_MODES) {
|
||||||
|
const entry = travelTimeEntries[mode];
|
||||||
|
if (entry?.destination) {
|
||||||
|
points.push({ position: [entry.destination[1], entry.destination[0]] });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return points;
|
||||||
|
}, [travelTimeEntries]);
|
||||||
|
|
||||||
const destinationMarkerLayer = useMemo(() => {
|
const destinationMarkerLayer = useMemo(() => {
|
||||||
if (!travelTimeEnabled || !travelTimeDestination) return null;
|
if (destinationMarkerData.length === 0) return null;
|
||||||
return new ScatterplotLayer({
|
return new ScatterplotLayer({
|
||||||
id: 'travel-time-destination',
|
id: 'travel-time-destinations',
|
||||||
data: [{ position: [travelTimeDestination[1], travelTimeDestination[0]] }],
|
data: destinationMarkerData,
|
||||||
getPosition: (d: { position: [number, number] }) => d.position,
|
getPosition: (d: { position: [number, number] }) => d.position,
|
||||||
getRadius: 8,
|
getRadius: 8,
|
||||||
getFillColor: [239, 68, 68, 220],
|
getFillColor: [239, 68, 68, 220],
|
||||||
|
|
@ -511,14 +559,14 @@ export function useDeckLayers({
|
||||||
stroked: true,
|
stroked: true,
|
||||||
pickable: false,
|
pickable: false,
|
||||||
});
|
});
|
||||||
}, [travelTimeEnabled, travelTimeDestination]);
|
}, [destinationMarkerData]);
|
||||||
|
|
||||||
const layers = useMemo(() => {
|
const layers = useMemo(() => {
|
||||||
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
// eslint-disable-next-line @typescript-eslint/no-explicit-any
|
||||||
const baseLayers: any[] = usePostcodeView
|
const baseLayers: any[] = usePostcodeView
|
||||||
? [postcodeLayer, postcodeLabelsLayer, poiLayer]
|
? [postcodeLayer, postcodeLabelsLayer, poiLayer]
|
||||||
: [hexLayer, poiLayer];
|
: [hexLayer, poiLayer];
|
||||||
if (searchedPostcodeHighlightLayer) baseLayers.push(searchedPostcodeHighlightLayer);
|
if (selectedPostcodeHighlightLayer) baseLayers.push(selectedPostcodeHighlightLayer);
|
||||||
if (destinationMarkerLayer) baseLayers.push(destinationMarkerLayer);
|
if (destinationMarkerLayer) baseLayers.push(destinationMarkerLayer);
|
||||||
return baseLayers;
|
return baseLayers;
|
||||||
}, [
|
}, [
|
||||||
|
|
@ -527,7 +575,7 @@ export function useDeckLayers({
|
||||||
postcodeLayer,
|
postcodeLayer,
|
||||||
postcodeLabelsLayer,
|
postcodeLabelsLayer,
|
||||||
poiLayer,
|
poiLayer,
|
||||||
searchedPostcodeHighlightLayer,
|
selectedPostcodeHighlightLayer,
|
||||||
destinationMarkerLayer,
|
destinationMarkerLayer,
|
||||||
]);
|
]);
|
||||||
|
|
||||||
|
|
@ -548,5 +596,6 @@ export function useDeckLayers({
|
||||||
handleMouseLeave,
|
handleMouseLeave,
|
||||||
selectedPostcode,
|
selectedPostcode,
|
||||||
hoveredPostcode,
|
hoveredPostcode,
|
||||||
|
primaryTravelMode,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -78,7 +78,8 @@ export function useFilters({ initialFilters, features }: UseFiltersOptions) {
|
||||||
const m = features.find((f) => f.name === n);
|
const m = features.find((f) => f.name === n);
|
||||||
if (m?.type === 'enum') return `${n}:${(value as string[]).join('|')}`;
|
if (m?.type === 'enum') return `${n}:${(value as string[]).join('|')}`;
|
||||||
const [min, max] = value as [number, number];
|
const [min, max] = value as [number, number];
|
||||||
return `${n}:${min}:${max}`;
|
const maxStr = m?.absolute && max === m.max ? 'inf' : String(max);
|
||||||
|
return `${n}:${min}:${maxStr}`;
|
||||||
})
|
})
|
||||||
.join(',');
|
.join(',');
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -3,10 +3,11 @@ import type {
|
||||||
FeatureMeta,
|
FeatureMeta,
|
||||||
FeatureFilters,
|
FeatureFilters,
|
||||||
Property,
|
Property,
|
||||||
|
PostcodeGeometry,
|
||||||
HexagonPropertiesResponse,
|
HexagonPropertiesResponse,
|
||||||
HexagonStatsResponse,
|
HexagonStatsResponse,
|
||||||
} from '../types';
|
} from '../types';
|
||||||
import { buildFilterString, apiUrl, logNonAbortError, authHeaders } from '../lib/api';
|
import { buildFilterString, apiUrl, assertOk, logNonAbortError, authHeaders } from '../lib/api';
|
||||||
|
|
||||||
interface SelectedHexagon {
|
interface SelectedHexagon {
|
||||||
id: string;
|
id: string;
|
||||||
|
|
@ -30,6 +31,8 @@ export function useHexagonSelection({ filters, features, resolution }: UseHexago
|
||||||
const [loadingAreaStats, setLoadingAreaStats] = useState(false);
|
const [loadingAreaStats, setLoadingAreaStats] = useState(false);
|
||||||
const [hoveredHexagon, setHoveredHexagon] = useState<string | null>(null);
|
const [hoveredHexagon, setHoveredHexagon] = useState<string | null>(null);
|
||||||
const [rightPaneTab, setRightPaneTab] = useState<'properties' | 'area'>('area');
|
const [rightPaneTab, setRightPaneTab] = useState<'properties' | 'area'>('area');
|
||||||
|
const [selectedPostcodeGeometry, setSelectedPostcodeGeometry] =
|
||||||
|
useState<PostcodeGeometry | null>(null);
|
||||||
|
|
||||||
const fetchHexagonStats = useCallback(
|
const fetchHexagonStats = useCallback(
|
||||||
async (h3: string, res: number, signal?: AbortSignal, fields?: string[]) => {
|
async (h3: string, res: number, signal?: AbortSignal, fields?: string[]) => {
|
||||||
|
|
@ -43,6 +46,7 @@ export function useHexagonSelection({ filters, features, resolution }: UseHexago
|
||||||
params.set('fields', fields.join(','));
|
params.set('fields', fields.join(','));
|
||||||
}
|
}
|
||||||
const response = await fetch(apiUrl('hexagon-stats', params), authHeaders({ signal }));
|
const response = await fetch(apiUrl('hexagon-stats', params), authHeaders({ signal }));
|
||||||
|
assertOk(response, 'hexagon-stats');
|
||||||
return (await response.json()) as HexagonStatsResponse;
|
return (await response.json()) as HexagonStatsResponse;
|
||||||
},
|
},
|
||||||
[filters, features]
|
[filters, features]
|
||||||
|
|
@ -54,6 +58,7 @@ export function useHexagonSelection({ filters, features, resolution }: UseHexago
|
||||||
const filterStr = buildFilterString(filters, features);
|
const filterStr = buildFilterString(filters, features);
|
||||||
if (filterStr) params.append('filters', filterStr);
|
if (filterStr) params.append('filters', filterStr);
|
||||||
const response = await fetch(apiUrl('postcode-stats', params), authHeaders({ signal }));
|
const response = await fetch(apiUrl('postcode-stats', params), authHeaders({ signal }));
|
||||||
|
assertOk(response, 'postcode-stats');
|
||||||
return (await response.json()) as HexagonStatsResponse;
|
return (await response.json()) as HexagonStatsResponse;
|
||||||
},
|
},
|
||||||
[filters, features]
|
[filters, features]
|
||||||
|
|
@ -74,6 +79,7 @@ export function useHexagonSelection({ filters, features, resolution }: UseHexago
|
||||||
if (filterStr) params.append('filters', filterStr);
|
if (filterStr) params.append('filters', filterStr);
|
||||||
|
|
||||||
const response = await fetch(apiUrl('hexagon-properties', params), authHeaders());
|
const response = await fetch(apiUrl('hexagon-properties', params), authHeaders());
|
||||||
|
assertOk(response, 'hexagon-properties');
|
||||||
const data: HexagonPropertiesResponse = await response.json();
|
const data: HexagonPropertiesResponse = await response.json();
|
||||||
|
|
||||||
if (offset === 0) {
|
if (offset === 0) {
|
||||||
|
|
@ -84,7 +90,7 @@ export function useHexagonSelection({ filters, features, resolution }: UseHexago
|
||||||
setPropertiesTotal(data.total);
|
setPropertiesTotal(data.total);
|
||||||
setPropertiesOffset(offset + data.properties.length);
|
setPropertiesOffset(offset + data.properties.length);
|
||||||
} catch (err) {
|
} catch (err) {
|
||||||
console.error('Failed to fetch properties:', err);
|
logNonAbortError('Failed to fetch properties', err);
|
||||||
} finally {
|
} finally {
|
||||||
setLoadingProperties(false);
|
setLoadingProperties(false);
|
||||||
}
|
}
|
||||||
|
|
@ -94,6 +100,7 @@ export function useHexagonSelection({ filters, features, resolution }: UseHexago
|
||||||
|
|
||||||
const handleHexagonClick = useCallback(
|
const handleHexagonClick = useCallback(
|
||||||
(id: string, isPostcode = false) => {
|
(id: string, isPostcode = false) => {
|
||||||
|
setSelectedPostcodeGeometry(null);
|
||||||
if (selectedHexagon?.id === id) {
|
if (selectedHexagon?.id === id) {
|
||||||
setSelectedHexagon(null);
|
setSelectedHexagon(null);
|
||||||
setProperties([]);
|
setProperties([]);
|
||||||
|
|
@ -154,8 +161,27 @@ export function useHexagonSelection({ filters, features, resolution }: UseHexago
|
||||||
setSelectedHexagon(null);
|
setSelectedHexagon(null);
|
||||||
setProperties([]);
|
setProperties([]);
|
||||||
setAreaStats(null);
|
setAreaStats(null);
|
||||||
|
setSelectedPostcodeGeometry(null);
|
||||||
}, []);
|
}, []);
|
||||||
|
|
||||||
|
const handleLocationSearch = useCallback(
|
||||||
|
(postcode: string, geometry: PostcodeGeometry) => {
|
||||||
|
setSelectedHexagon({ id: postcode, type: 'postcode', resolution });
|
||||||
|
setSelectedPostcodeGeometry(geometry);
|
||||||
|
setProperties([]);
|
||||||
|
setPropertiesTotal(0);
|
||||||
|
setPropertiesOffset(0);
|
||||||
|
setRightPaneTab('area');
|
||||||
|
|
||||||
|
setLoadingAreaStats(true);
|
||||||
|
fetchPostcodeStats(postcode)
|
||||||
|
.then((stats) => setAreaStats(stats))
|
||||||
|
.catch((error) => logNonAbortError('Failed to fetch postcode stats', error))
|
||||||
|
.finally(() => setLoadingAreaStats(false));
|
||||||
|
},
|
||||||
|
[resolution, fetchPostcodeStats]
|
||||||
|
);
|
||||||
|
|
||||||
return {
|
return {
|
||||||
selectedHexagon,
|
selectedHexagon,
|
||||||
properties,
|
properties,
|
||||||
|
|
@ -172,5 +198,7 @@ export function useHexagonSelection({ filters, features, resolution }: UseHexago
|
||||||
handlePropertiesTabClick,
|
handlePropertiesTabClick,
|
||||||
handleLoadMoreProperties,
|
handleLoadMoreProperties,
|
||||||
handleCloseSelection,
|
handleCloseSelection,
|
||||||
|
selectedPostcodeGeometry,
|
||||||
|
handleLocationSearch,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
|
||||||
123
frontend/src/hooks/useLocationSearch.ts
Normal file
123
frontend/src/hooks/useLocationSearch.ts
Normal file
|
|
@ -0,0 +1,123 @@
|
||||||
|
import { useState, useCallback, useRef, useEffect } from 'react';
|
||||||
|
import type { PlaceResult } from '../types';
|
||||||
|
import { authHeaders, logNonAbortError } from '../lib/api';
|
||||||
|
|
||||||
|
const POSTCODE_RE = /^[A-Z]{1,2}\d[A-Z\d]?\s*\d?[A-Z]{0,2}$/i;
|
||||||
|
|
||||||
|
export function looksLikePostcode(s: string) {
|
||||||
|
return POSTCODE_RE.test(s.trim());
|
||||||
|
}
|
||||||
|
|
||||||
|
export type SearchResult =
|
||||||
|
| { type: 'postcode'; label: string }
|
||||||
|
| { type: 'place'; name: string; place_type: string; lat: number; lon: number; city?: string };
|
||||||
|
|
||||||
|
export function useLocationSearch() {
|
||||||
|
const [query, setQuery] = useState('');
|
||||||
|
const [results, setResults] = useState<SearchResult[]>([]);
|
||||||
|
const [activeIndex, setActiveIndex] = useState(-1);
|
||||||
|
const [open, setOpen] = useState(false);
|
||||||
|
const abortRef = useRef<AbortController | null>(null);
|
||||||
|
const debounceRef = useRef<ReturnType<typeof setTimeout>>();
|
||||||
|
|
||||||
|
const handleInputChange = useCallback((value: string) => {
|
||||||
|
setQuery(value);
|
||||||
|
setActiveIndex(-1);
|
||||||
|
|
||||||
|
abortRef.current?.abort();
|
||||||
|
if (debounceRef.current) clearTimeout(debounceRef.current);
|
||||||
|
|
||||||
|
const trimmed = value.trim();
|
||||||
|
if (!trimmed) {
|
||||||
|
setResults([]);
|
||||||
|
setOpen(false);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (looksLikePostcode(trimmed)) {
|
||||||
|
setResults([{ type: 'postcode', label: trimmed.toUpperCase() }]);
|
||||||
|
setOpen(true);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (trimmed.length < 2) {
|
||||||
|
setResults([]);
|
||||||
|
setOpen(false);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
debounceRef.current = setTimeout(async () => {
|
||||||
|
const controller = new AbortController();
|
||||||
|
abortRef.current = controller;
|
||||||
|
try {
|
||||||
|
const params = new URLSearchParams({ q: trimmed, limit: '7' });
|
||||||
|
const res = await fetch(
|
||||||
|
`/api/places?${params}`,
|
||||||
|
authHeaders({ signal: controller.signal }),
|
||||||
|
);
|
||||||
|
if (!res.ok) return;
|
||||||
|
const json: { places: PlaceResult[] } = await res.json();
|
||||||
|
const placeResults: SearchResult[] = json.places.map((p) => ({
|
||||||
|
type: 'place' as const,
|
||||||
|
...p,
|
||||||
|
}));
|
||||||
|
setResults(placeResults);
|
||||||
|
setOpen(placeResults.length > 0);
|
||||||
|
} catch (err) {
|
||||||
|
logNonAbortError('places search', err);
|
||||||
|
}
|
||||||
|
}, 200);
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const close = useCallback(() => setOpen(false), []);
|
||||||
|
|
||||||
|
const clear = useCallback(() => {
|
||||||
|
setQuery('');
|
||||||
|
setResults([]);
|
||||||
|
setOpen(false);
|
||||||
|
setActiveIndex(-1);
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
const handleKeyDown = useCallback(
|
||||||
|
(e: React.KeyboardEvent, onSelect: (result: SearchResult) => void) => {
|
||||||
|
if (e.key === 'ArrowDown') {
|
||||||
|
e.preventDefault();
|
||||||
|
setActiveIndex((prev) => (prev < results.length - 1 ? prev + 1 : prev));
|
||||||
|
} else if (e.key === 'ArrowUp') {
|
||||||
|
e.preventDefault();
|
||||||
|
setActiveIndex((prev) => (prev > 0 ? prev - 1 : -1));
|
||||||
|
} else if (e.key === 'Enter') {
|
||||||
|
e.preventDefault();
|
||||||
|
if (activeIndex >= 0 && activeIndex < results.length) {
|
||||||
|
onSelect(results[activeIndex]);
|
||||||
|
} else if (looksLikePostcode(query)) {
|
||||||
|
onSelect({ type: 'postcode', label: query.trim().toUpperCase() });
|
||||||
|
}
|
||||||
|
} else if (e.key === 'Escape') {
|
||||||
|
setOpen(false);
|
||||||
|
}
|
||||||
|
},
|
||||||
|
[results, activeIndex, query],
|
||||||
|
);
|
||||||
|
|
||||||
|
// Cleanup on unmount
|
||||||
|
useEffect(() => {
|
||||||
|
return () => {
|
||||||
|
abortRef.current?.abort();
|
||||||
|
if (debounceRef.current) clearTimeout(debounceRef.current);
|
||||||
|
};
|
||||||
|
}, []);
|
||||||
|
|
||||||
|
return {
|
||||||
|
query,
|
||||||
|
results,
|
||||||
|
activeIndex,
|
||||||
|
setActiveIndex,
|
||||||
|
open,
|
||||||
|
setOpen,
|
||||||
|
handleInputChange,
|
||||||
|
handleKeyDown,
|
||||||
|
close,
|
||||||
|
clear,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
@ -8,9 +8,10 @@ import type {
|
||||||
ViewChangeParams,
|
ViewChangeParams,
|
||||||
ApiResponse,
|
ApiResponse,
|
||||||
} from '../types';
|
} from '../types';
|
||||||
import { buildFilterString, apiUrl, logNonAbortError, authHeaders } from '../lib/api';
|
import { buildFilterString, apiUrl, assertOk, logNonAbortError, authHeaders } from '../lib/api';
|
||||||
import { POSTCODE_ZOOM_THRESHOLD } from '../lib/consts';
|
import { POSTCODE_ZOOM_THRESHOLD } from '../lib/consts';
|
||||||
import { COLOR_RANGE_LOW_PERCENTILE, COLOR_RANGE_HIGH_PERCENTILE } from '../lib/consts';
|
import { COLOR_RANGE_LOW_PERCENTILE, COLOR_RANGE_HIGH_PERCENTILE } from '../lib/consts';
|
||||||
|
import { TRANSPORT_MODES, type TransportMode, type TravelTimeEntries } from './useTravelTime';
|
||||||
|
|
||||||
/** Return the p-th percentile (0–100) from a sorted array via linear interpolation. */
|
/** Return the p-th percentile (0–100) from a sorted array via linear interpolation. */
|
||||||
function percentile(sorted: number[], p: number): number {
|
function percentile(sorted: number[], p: number): number {
|
||||||
|
|
@ -32,9 +33,7 @@ interface UseMapDataOptions {
|
||||||
activeFeature: string | null;
|
activeFeature: string | null;
|
||||||
dragValue: [number, number] | null;
|
dragValue: [number, number] | null;
|
||||||
dragData: HexagonData[] | null;
|
dragData: HexagonData[] | null;
|
||||||
travelTimeEnabled: boolean;
|
travelTimeEntries: TravelTimeEntries;
|
||||||
travelTimeDestination: [number, number] | null;
|
|
||||||
travelTimeMode: string;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
export function useMapData({
|
export function useMapData({
|
||||||
|
|
@ -44,9 +43,7 @@ export function useMapData({
|
||||||
activeFeature,
|
activeFeature,
|
||||||
dragValue,
|
dragValue,
|
||||||
dragData,
|
dragData,
|
||||||
travelTimeEnabled,
|
travelTimeEntries,
|
||||||
travelTimeDestination,
|
|
||||||
travelTimeMode,
|
|
||||||
}: UseMapDataOptions) {
|
}: UseMapDataOptions) {
|
||||||
const [rawData, setRawData] = useState<HexagonData[]>([]);
|
const [rawData, setRawData] = useState<HexagonData[]>([]);
|
||||||
const [postcodeData, setPostcodeData] = useState<PostcodeFeature[]>([]);
|
const [postcodeData, setPostcodeData] = useState<PostcodeFeature[]>([]);
|
||||||
|
|
@ -71,6 +68,18 @@ export function useMapData({
|
||||||
[filters, features]
|
[filters, features]
|
||||||
);
|
);
|
||||||
|
|
||||||
|
// Build the travel param string from entries with destinations
|
||||||
|
const travelParam = useMemo((): string => {
|
||||||
|
const segments: string[] = [];
|
||||||
|
for (const mode of TRANSPORT_MODES) {
|
||||||
|
const entry = travelTimeEntries[mode];
|
||||||
|
if (entry?.destination) {
|
||||||
|
segments.push(`${entry.destination[0]},${entry.destination[1]},${mode}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return segments.join('|');
|
||||||
|
}, [travelTimeEntries]);
|
||||||
|
|
||||||
// Fetch hexagons or postcodes when bounds/filters change
|
// Fetch hexagons or postcodes when bounds/filters change
|
||||||
useEffect(() => {
|
useEffect(() => {
|
||||||
if (!bounds) return;
|
if (!bounds) return;
|
||||||
|
|
@ -100,6 +109,7 @@ export function useMapData({
|
||||||
signal: abortControllerRef.current.signal,
|
signal: abortControllerRef.current.signal,
|
||||||
})
|
})
|
||||||
);
|
);
|
||||||
|
assertOk(res, 'postcodes');
|
||||||
const json: { features: PostcodeFeature[] } = await res.json();
|
const json: { features: PostcodeFeature[] } = await res.json();
|
||||||
setPostcodeData(json.features);
|
setPostcodeData(json.features);
|
||||||
setRawData([]);
|
setRawData([]);
|
||||||
|
|
@ -110,9 +120,8 @@ export function useMapData({
|
||||||
});
|
});
|
||||||
if (filtersStr) params.set('filters', filtersStr);
|
if (filtersStr) params.set('filters', filtersStr);
|
||||||
params.set('fields', viewFeature || '');
|
params.set('fields', viewFeature || '');
|
||||||
if (travelTimeEnabled && travelTimeDestination) {
|
if (travelParam) {
|
||||||
params.set('destination', `${travelTimeDestination[0]},${travelTimeDestination[1]}`);
|
params.set('travel', travelParam);
|
||||||
params.set('mode', travelTimeMode);
|
|
||||||
}
|
}
|
||||||
const res = await fetch(
|
const res = await fetch(
|
||||||
apiUrl('hexagons', params),
|
apiUrl('hexagons', params),
|
||||||
|
|
@ -120,6 +129,7 @@ export function useMapData({
|
||||||
signal: abortControllerRef.current.signal,
|
signal: abortControllerRef.current.signal,
|
||||||
})
|
})
|
||||||
);
|
);
|
||||||
|
assertOk(res, 'hexagons');
|
||||||
const json: ApiResponse = await res.json();
|
const json: ApiResponse = await res.json();
|
||||||
setRawData(json.features);
|
setRawData(json.features);
|
||||||
setPostcodeData([]);
|
setPostcodeData([]);
|
||||||
|
|
@ -136,7 +146,7 @@ export function useMapData({
|
||||||
clearTimeout(debounceRef.current);
|
clearTimeout(debounceRef.current);
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
}, [resolution, bounds, filters, buildFilterParam, viewFeature, usePostcodeView, travelTimeEnabled, travelTimeDestination, travelTimeMode]);
|
}, [resolution, bounds, filters, buildFilterParam, viewFeature, usePostcodeView, travelParam]);
|
||||||
|
|
||||||
const data = dragData ?? rawData;
|
const data = dragData ?? rawData;
|
||||||
|
|
||||||
|
|
@ -159,7 +169,7 @@ export function useMapData({
|
||||||
if (lat < bounds.south || lat > bounds.north || lng < bounds.west || lng > bounds.east)
|
if (lat < bounds.south || lat > bounds.north || lng < bounds.west || lng > bounds.east)
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
const val = feat.properties[`avg_${viewFeature}`] ?? feat.properties[`min_${viewFeature}`];
|
const val = feat.properties[`avg_${viewFeature}`];
|
||||||
if (typeof val === 'number' && !isNaN(val)) vals.push(val);
|
if (typeof val === 'number' && !isNaN(val)) vals.push(val);
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
|
|
@ -170,7 +180,7 @@ export function useMapData({
|
||||||
if (lat < bounds.south || lat > bounds.north || lon < bounds.west || lon > bounds.east)
|
if (lat < bounds.south || lat > bounds.north || lon < bounds.west || lon > bounds.east)
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
const val = item[`avg_${viewFeature}`] ?? item[`min_${viewFeature}`];
|
const val = item[`avg_${viewFeature}`];
|
||||||
if (typeof val === 'number' && !isNaN(val)) vals.push(val);
|
if (typeof val === 'number' && !isNaN(val)) vals.push(val);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -197,26 +207,32 @@ export function useMapData({
|
||||||
return null;
|
return null;
|
||||||
}, [viewFeature, features, dataRange, activeFeature, dragValue]);
|
}, [viewFeature, features, dataRange, activeFeature, dragValue]);
|
||||||
|
|
||||||
// Color range for travel time (computed from response data)
|
// Color ranges for travel time per mode (computed from response data)
|
||||||
const travelTimeColorRange = useMemo((): [number, number] | null => {
|
const travelTimeColorRanges = useMemo((): Partial<Record<TransportMode, [number, number]>> => {
|
||||||
if (!travelTimeEnabled || !travelTimeDestination) return null;
|
const ranges: Partial<Record<TransportMode, [number, number]>> = {};
|
||||||
const vals: number[] = [];
|
for (const mode of TRANSPORT_MODES) {
|
||||||
for (const item of data) {
|
const entry = travelTimeEntries[mode];
|
||||||
if (bounds) {
|
if (!entry?.destination) continue;
|
||||||
const { lat, lon } = item;
|
const fieldName = `travel_time_${mode}`;
|
||||||
if (lat < bounds.south || lat > bounds.north || lon < bounds.west || lon > bounds.east)
|
const vals: number[] = [];
|
||||||
continue;
|
for (const item of data) {
|
||||||
|
if (bounds) {
|
||||||
|
const { lat, lon } = item;
|
||||||
|
if (lat < bounds.south || lat > bounds.north || lon < bounds.west || lon > bounds.east)
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
const val = item[fieldName];
|
||||||
|
if (typeof val === 'number' && !isNaN(val)) vals.push(val);
|
||||||
}
|
}
|
||||||
const val = item.travel_time;
|
if (vals.length === 0) continue;
|
||||||
if (typeof val === 'number' && !isNaN(val)) vals.push(val);
|
vals.sort((a, b) => a - b);
|
||||||
|
ranges[mode] = [
|
||||||
|
percentile(vals, COLOR_RANGE_LOW_PERCENTILE),
|
||||||
|
percentile(vals, COLOR_RANGE_HIGH_PERCENTILE),
|
||||||
|
];
|
||||||
}
|
}
|
||||||
if (vals.length === 0) return null;
|
return ranges;
|
||||||
vals.sort((a, b) => a - b);
|
}, [travelTimeEntries, data, bounds]);
|
||||||
return [
|
|
||||||
percentile(vals, COLOR_RANGE_LOW_PERCENTILE),
|
|
||||||
percentile(vals, COLOR_RANGE_HIGH_PERCENTILE),
|
|
||||||
];
|
|
||||||
}, [travelTimeEnabled, travelTimeDestination, data, bounds]);
|
|
||||||
|
|
||||||
const handleViewChange = useCallback(
|
const handleViewChange = useCallback(
|
||||||
({
|
({
|
||||||
|
|
@ -257,7 +273,7 @@ export function useMapData({
|
||||||
currentView,
|
currentView,
|
||||||
usePostcodeView,
|
usePostcodeView,
|
||||||
colorRange,
|
colorRange,
|
||||||
travelTimeColorRange,
|
travelTimeColorRanges,
|
||||||
handleViewChange,
|
handleViewChange,
|
||||||
setInitialView,
|
setInitialView,
|
||||||
};
|
};
|
||||||
|
|
|
||||||
|
|
@ -1,67 +1,83 @@
|
||||||
import { useState, useCallback } from 'react';
|
import { useState, useCallback, useMemo } from 'react';
|
||||||
|
|
||||||
export type TransportMode = 'car' | 'bicycle' | 'walking' | 'transit';
|
export type TransportMode = 'car' | 'bicycle' | 'walking' | 'transit';
|
||||||
|
|
||||||
export interface TravelTimeState {
|
export const TRANSPORT_MODES: TransportMode[] = ['car', 'bicycle', 'walking', 'transit'];
|
||||||
enabled: boolean;
|
|
||||||
|
export const MODE_LABELS: Record<TransportMode, string> = {
|
||||||
|
car: 'Car',
|
||||||
|
bicycle: 'Bicycle',
|
||||||
|
walking: 'Walking',
|
||||||
|
transit: 'Transit',
|
||||||
|
};
|
||||||
|
|
||||||
|
export interface TravelTimeEntry {
|
||||||
destination: [number, number] | null; // [lat, lon]
|
destination: [number, number] | null; // [lat, lon]
|
||||||
destinationLabel: string;
|
destinationLabel: string;
|
||||||
mode: TransportMode;
|
|
||||||
timeRange: [number, number] | null;
|
timeRange: [number, number] | null;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export type TravelTimeEntries = Partial<Record<TransportMode, TravelTimeEntry>>;
|
||||||
|
|
||||||
export interface TravelTimeInitial {
|
export interface TravelTimeInitial {
|
||||||
destination?: [number, number];
|
entries?: TravelTimeEntries;
|
||||||
destinationLabel?: string;
|
|
||||||
mode?: TransportMode;
|
|
||||||
timeRange?: [number, number];
|
|
||||||
}
|
}
|
||||||
|
|
||||||
export function useTravelTime(initial?: TravelTimeInitial) {
|
export function useTravelTime(initial?: TravelTimeInitial) {
|
||||||
const [enabled, setEnabled] = useState(!!initial?.destination);
|
const [entries, setEntries] = useState<TravelTimeEntries>(initial?.entries ?? {});
|
||||||
const [destination, setDestination] = useState<[number, number] | null>(
|
|
||||||
initial?.destination ?? null
|
const activeModes = useMemo(
|
||||||
);
|
() => TRANSPORT_MODES.filter((m) => m in entries),
|
||||||
const [destinationLabel, setDestinationLabel] = useState(initial?.destinationLabel ?? '');
|
[entries]
|
||||||
const [mode, setMode] = useState<TransportMode>(initial?.mode ?? 'car');
|
|
||||||
const [timeRange, setTimeRange] = useState<[number, number] | null>(
|
|
||||||
initial?.timeRange ?? null
|
|
||||||
);
|
);
|
||||||
|
|
||||||
const handleEnable = useCallback(() => {
|
const modesWithDestination = useMemo(
|
||||||
setEnabled(true);
|
() => TRANSPORT_MODES.filter((m) => entries[m]?.destination != null),
|
||||||
|
[entries]
|
||||||
|
);
|
||||||
|
|
||||||
|
const handleEnableMode = useCallback((mode: TransportMode) => {
|
||||||
|
setEntries((prev) => ({
|
||||||
|
...prev,
|
||||||
|
[mode]: { destination: null, destinationLabel: '', timeRange: null },
|
||||||
|
}));
|
||||||
}, []);
|
}, []);
|
||||||
|
|
||||||
const handleDisable = useCallback(() => {
|
const handleDisableMode = useCallback((mode: TransportMode) => {
|
||||||
setEnabled(false);
|
setEntries((prev) => {
|
||||||
setDestination(null);
|
const next = { ...prev };
|
||||||
setDestinationLabel('');
|
delete next[mode];
|
||||||
setTimeRange(null);
|
return next;
|
||||||
|
});
|
||||||
}, []);
|
}, []);
|
||||||
|
|
||||||
const handleSetDestination = useCallback((lat: number, lon: number, label: string) => {
|
const handleSetDestination = useCallback(
|
||||||
setDestination([lat, lon]);
|
(mode: TransportMode, lat: number, lon: number, label: string) => {
|
||||||
setDestinationLabel(label);
|
setEntries((prev) => ({
|
||||||
}, []);
|
...prev,
|
||||||
|
[mode]: { ...prev[mode], destination: [lat, lon] as [number, number], destinationLabel: label },
|
||||||
|
}));
|
||||||
|
},
|
||||||
|
[]
|
||||||
|
);
|
||||||
|
|
||||||
const handleModeChange = useCallback((newMode: TransportMode) => {
|
const handleTimeRangeChange = useCallback(
|
||||||
setMode(newMode);
|
(mode: TransportMode, range: [number, number]) => {
|
||||||
}, []);
|
setEntries((prev) => ({
|
||||||
|
...prev,
|
||||||
const handleTimeRangeChange = useCallback((range: [number, number]) => {
|
[mode]: { ...prev[mode], timeRange: range },
|
||||||
setTimeRange(range);
|
}));
|
||||||
}, []);
|
},
|
||||||
|
[]
|
||||||
|
);
|
||||||
|
|
||||||
return {
|
return {
|
||||||
enabled,
|
entries,
|
||||||
destination,
|
activeModes,
|
||||||
destinationLabel,
|
modesWithDestination,
|
||||||
mode,
|
handleEnableMode,
|
||||||
timeRange,
|
handleDisableMode,
|
||||||
handleEnable,
|
|
||||||
handleDisable,
|
|
||||||
handleSetDestination,
|
handleSetDestination,
|
||||||
handleModeChange,
|
|
||||||
handleTimeRangeChange,
|
handleTimeRangeChange,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,15 +1,7 @@
|
||||||
import { useEffect, useRef } from 'react';
|
import { useEffect, useRef } from 'react';
|
||||||
import type { FeatureMeta, FeatureFilters } from '../types';
|
import type { FeatureMeta, FeatureFilters } from '../types';
|
||||||
import { stateToParams } from '../lib/url-state';
|
import { stateToParams } from '../lib/url-state';
|
||||||
import type { TransportMode } from './useTravelTime';
|
import type { TravelTimeEntries } from './useTravelTime';
|
||||||
|
|
||||||
export interface TravelTimeUrlState {
|
|
||||||
enabled: boolean;
|
|
||||||
destination: [number, number] | null;
|
|
||||||
destinationLabel: string;
|
|
||||||
mode: TransportMode;
|
|
||||||
timeRange: [number, number] | null;
|
|
||||||
}
|
|
||||||
|
|
||||||
const URL_DEBOUNCE_MS = 300;
|
const URL_DEBOUNCE_MS = 300;
|
||||||
|
|
||||||
|
|
@ -19,7 +11,7 @@ export function useUrlSync(
|
||||||
features: FeatureMeta[],
|
features: FeatureMeta[],
|
||||||
selectedPOICategories: Set<string>,
|
selectedPOICategories: Set<string>,
|
||||||
rightPaneTab: 'properties' | 'area',
|
rightPaneTab: 'properties' | 'area',
|
||||||
travelTime?: TravelTimeUrlState
|
travelTimeEntries?: TravelTimeEntries
|
||||||
) {
|
) {
|
||||||
const urlDebounceRef = useRef<ReturnType<typeof setTimeout> | null>(null);
|
const urlDebounceRef = useRef<ReturnType<typeof setTimeout> | null>(null);
|
||||||
|
|
||||||
|
|
@ -34,7 +26,7 @@ export function useUrlSync(
|
||||||
features,
|
features,
|
||||||
selectedPOICategories,
|
selectedPOICategories,
|
||||||
rightPaneTab,
|
rightPaneTab,
|
||||||
travelTime
|
travelTimeEntries
|
||||||
);
|
);
|
||||||
const search = params.toString();
|
const search = params.toString();
|
||||||
const newUrl = search ? `${window.location.pathname}?${search}` : window.location.pathname;
|
const newUrl = search ? `${window.location.pathname}?${search}` : window.location.pathname;
|
||||||
|
|
@ -44,5 +36,5 @@ export function useUrlSync(
|
||||||
return () => {
|
return () => {
|
||||||
if (urlDebounceRef.current) clearTimeout(urlDebounceRef.current);
|
if (urlDebounceRef.current) clearTimeout(urlDebounceRef.current);
|
||||||
};
|
};
|
||||||
}, [currentView, filters, features, selectedPOICategories, rightPaneTab, travelTime]);
|
}, [currentView, filters, features, selectedPOICategories, rightPaneTab, travelTimeEntries]);
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -10,6 +10,17 @@ export function logNonAbortError(label: string, error: unknown): void {
|
||||||
console.error(`${label}:`, error);
|
console.error(`${label}:`, error);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
export function isAbortError(error: unknown): boolean {
|
||||||
|
return error instanceof Error && error.name === 'AbortError';
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Throw if response is not 2xx. Call before `.json()`. */
|
||||||
|
export function assertOk(res: Response, label: string): void {
|
||||||
|
if (!res.ok) {
|
||||||
|
throw new Error(`${label}: HTTP ${res.status} ${res.statusText}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
export function authHeaders(init?: RequestInit): RequestInit {
|
export function authHeaders(init?: RequestInit): RequestInit {
|
||||||
const headers: Record<string, string> = {};
|
const headers: Record<string, string> = {};
|
||||||
if (pb.authStore.isValid && pb.authStore.token) {
|
if (pb.authStore.isValid && pb.authStore.token) {
|
||||||
|
|
@ -69,7 +80,8 @@ export function buildFilterString(filters: FeatureFilters, features: FeatureMeta
|
||||||
return `${name}:${(value as string[]).join('|')}`;
|
return `${name}:${(value as string[]).join('|')}`;
|
||||||
}
|
}
|
||||||
const [min, max] = value as [number, number];
|
const [min, max] = value as [number, number];
|
||||||
return `${name}:${min}:${max}`;
|
const maxStr = meta?.absolute && max === meta.max ? 'inf' : String(max);
|
||||||
|
return `${name}:${min}:${maxStr}`;
|
||||||
})
|
})
|
||||||
.join(',');
|
.join(',');
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,10 @@
|
||||||
import type { FeatureMeta, FeatureFilters, ViewState } from '../types';
|
import type { FeatureMeta, FeatureFilters, ViewState } from '../types';
|
||||||
import type { TransportMode, TravelTimeInitial } from '../hooks/useTravelTime';
|
import {
|
||||||
|
TRANSPORT_MODES,
|
||||||
|
type TransportMode,
|
||||||
|
type TravelTimeEntries,
|
||||||
|
type TravelTimeInitial,
|
||||||
|
} from '../hooks/useTravelTime';
|
||||||
|
|
||||||
function parseFilters(params: URLSearchParams): FeatureFilters | undefined {
|
function parseFilters(params: URLSearchParams): FeatureFilters | undefined {
|
||||||
const filterParams = params.getAll('filter');
|
const filterParams = params.getAll('filter');
|
||||||
|
|
@ -65,26 +70,33 @@ export function parseUrlState(): {
|
||||||
result.tab = tab;
|
result.tab = tab;
|
||||||
}
|
}
|
||||||
|
|
||||||
// Travel time
|
// Travel time: per-mode params (tt_car=lat,lon ttl_car=label ttr_car=min:max)
|
||||||
const dest = params.get('dest');
|
const entries: TravelTimeEntries = {};
|
||||||
if (dest) {
|
for (const mode of TRANSPORT_MODES) {
|
||||||
const parts = dest.split(',').map(Number);
|
const dest = params.get(`tt_${mode}`);
|
||||||
if (parts.length === 2 && parts.every((n) => !isNaN(n))) {
|
if (dest) {
|
||||||
const tt: TravelTimeInitial = {
|
const parts = dest.split(',').map(Number);
|
||||||
destination: [parts[0], parts[1]],
|
if (parts.length === 2 && parts.every((n) => !isNaN(n))) {
|
||||||
destinationLabel: params.get('destLabel') || '',
|
const label = params.get(`ttl_${mode}`) || '';
|
||||||
mode: (params.get('tmode') as TransportMode) || 'car',
|
let timeRange: [number, number] | null = null;
|
||||||
};
|
const rangeStr = params.get(`ttr_${mode}`);
|
||||||
const ttRange = params.get('tt');
|
if (rangeStr) {
|
||||||
if (ttRange) {
|
const [min, max] = rangeStr.split(':').map(Number);
|
||||||
const [min, max] = ttRange.split(':').map(Number);
|
if (!isNaN(min) && !isNaN(max)) {
|
||||||
if (!isNaN(min) && !isNaN(max)) {
|
timeRange = [min, max];
|
||||||
tt.timeRange = [min, max];
|
}
|
||||||
}
|
}
|
||||||
|
entries[mode] = {
|
||||||
|
destination: [parts[0], parts[1]],
|
||||||
|
destinationLabel: label,
|
||||||
|
timeRange,
|
||||||
|
};
|
||||||
}
|
}
|
||||||
result.travelTime = tt;
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if (Object.keys(entries).length > 0) {
|
||||||
|
result.travelTime = { entries };
|
||||||
|
}
|
||||||
|
|
||||||
return result;
|
return result;
|
||||||
}
|
}
|
||||||
|
|
@ -95,7 +107,7 @@ export function stateToParams(
|
||||||
features: FeatureMeta[],
|
features: FeatureMeta[],
|
||||||
selectedPOICategories: Set<string>,
|
selectedPOICategories: Set<string>,
|
||||||
rightPaneTab: 'properties' | 'area',
|
rightPaneTab: 'properties' | 'area',
|
||||||
travelTime?: { enabled: boolean; destination: [number, number] | null; destinationLabel: string; mode: string; timeRange: [number, number] | null }
|
travelTimeEntries?: TravelTimeEntries
|
||||||
): URLSearchParams {
|
): URLSearchParams {
|
||||||
const params = new URLSearchParams();
|
const params = new URLSearchParams();
|
||||||
|
|
||||||
|
|
@ -123,16 +135,18 @@ export function stateToParams(
|
||||||
params.set('tab', 'properties');
|
params.set('tab', 'properties');
|
||||||
}
|
}
|
||||||
|
|
||||||
if (travelTime?.enabled && travelTime.destination) {
|
// Travel time: per-mode params
|
||||||
params.set('dest', `${travelTime.destination[0].toFixed(5)},${travelTime.destination[1].toFixed(5)}`);
|
if (travelTimeEntries) {
|
||||||
if (travelTime.destinationLabel) {
|
for (const mode of TRANSPORT_MODES) {
|
||||||
params.set('destLabel', travelTime.destinationLabel);
|
const entry = travelTimeEntries[mode];
|
||||||
}
|
if (!entry?.destination) continue;
|
||||||
if (travelTime.mode !== 'car') {
|
params.set(`tt_${mode}`, `${entry.destination[0].toFixed(5)},${entry.destination[1].toFixed(5)}`);
|
||||||
params.set('tmode', travelTime.mode);
|
if (entry.destinationLabel) {
|
||||||
}
|
params.set(`ttl_${mode}`, entry.destinationLabel);
|
||||||
if (travelTime.timeRange) {
|
}
|
||||||
params.set('tt', `${travelTime.timeRange[0]}:${travelTime.timeRange[1]}`);
|
if (entry.timeRange) {
|
||||||
|
params.set(`ttr_${mode}`, `${entry.timeRange[0]}:${entry.timeRange[1]}`);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -17,6 +17,7 @@ export interface FeatureMeta {
|
||||||
prefix?: string;
|
prefix?: string;
|
||||||
suffix?: string;
|
suffix?: string;
|
||||||
raw?: boolean;
|
raw?: boolean;
|
||||||
|
absolute?: boolean;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface FeatureGroup {
|
export interface FeatureGroup {
|
||||||
|
|
@ -104,6 +105,7 @@ export interface PlaceResult {
|
||||||
place_type: string;
|
place_type: string;
|
||||||
lat: number;
|
lat: number;
|
||||||
lon: number;
|
lon: number;
|
||||||
|
city?: string;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface RenovationEvent {
|
export interface RenovationEvent {
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,7 @@
|
||||||
"""Extract place=* nodes from OSM PBF → data/places.parquet.
|
"""Extract place=* nodes and railway stations from OSM PBF → data/places.parquet.
|
||||||
|
|
||||||
Extracts named place nodes (cities, towns, suburbs, etc.) for typeahead search.
|
Extracts named place nodes (cities, towns, suburbs, etc.) and railway stations
|
||||||
|
(tube, national rail, DLR, etc.) for typeahead search.
|
||||||
Reuses the same great-britain-latest.osm.pbf as pois.py.
|
Reuses the same great-britain-latest.osm.pbf as pois.py.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
|
|
@ -18,13 +19,54 @@ PLACE_TYPES = {
|
||||||
"borough",
|
"borough",
|
||||||
"town",
|
"town",
|
||||||
"suburb",
|
"suburb",
|
||||||
|
"quarter",
|
||||||
"neighbourhood",
|
"neighbourhood",
|
||||||
"village",
|
"village",
|
||||||
"hamlet",
|
"hamlet",
|
||||||
"locality",
|
"locality",
|
||||||
|
"island",
|
||||||
"isolated_dwelling",
|
"isolated_dwelling",
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Suffixes to strip from raw station names before appending the typed suffix.
|
||||||
|
_STATION_STRIP = (
|
||||||
|
" tube station",
|
||||||
|
" underground station",
|
||||||
|
" railway station",
|
||||||
|
" dlr station",
|
||||||
|
" overground station",
|
||||||
|
" tram stop",
|
||||||
|
" station",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _station_display_name(name: str, tags: dict[str, str]) -> str:
|
||||||
|
"""Build a descriptive station name like 'Bank tube station'."""
|
||||||
|
station_tag = tags.get("station", "")
|
||||||
|
network = tags.get("network", "").lower()
|
||||||
|
|
||||||
|
if station_tag == "subway" or "underground" in network:
|
||||||
|
suffix = "tube station"
|
||||||
|
elif "docklands" in network or "dlr" in network:
|
||||||
|
suffix = "DLR station"
|
||||||
|
elif "overground" in network:
|
||||||
|
suffix = "overground station"
|
||||||
|
elif "elizabeth" in network:
|
||||||
|
suffix = "Elizabeth line station"
|
||||||
|
elif station_tag == "light_rail" or "tramlink" in network or "tram" in network:
|
||||||
|
suffix = "tram stop"
|
||||||
|
else:
|
||||||
|
suffix = "railway station"
|
||||||
|
|
||||||
|
# Strip any existing station suffix from the raw name
|
||||||
|
lower = name.lower()
|
||||||
|
for s in _STATION_STRIP:
|
||||||
|
if lower.endswith(s):
|
||||||
|
name = name[: len(name) - len(s)].rstrip()
|
||||||
|
break
|
||||||
|
|
||||||
|
return f"{name} {suffix}"
|
||||||
|
|
||||||
|
|
||||||
class PlaceHandler(osmium.SimpleHandler):
|
class PlaceHandler(osmium.SimpleHandler):
|
||||||
def __init__(self, progress: tqdm) -> None:
|
def __init__(self, progress: tqdm) -> None:
|
||||||
|
|
@ -32,6 +74,12 @@ class PlaceHandler(osmium.SimpleHandler):
|
||||||
self._progress = progress
|
self._progress = progress
|
||||||
self.places: list[dict] = []
|
self.places: list[dict] = []
|
||||||
|
|
||||||
|
def _add(self, name: str, place_type: str, lat: float, lon: float, population: int) -> None:
|
||||||
|
self.places.append(
|
||||||
|
{"name": name, "place_type": place_type, "lat": lat, "lon": lon, "population": population}
|
||||||
|
)
|
||||||
|
self._progress.set_postfix(places=f"{len(self.places):,}", refresh=False)
|
||||||
|
|
||||||
def node(self, n: osmium.osm.Node) -> None:
|
def node(self, n: osmium.osm.Node) -> None:
|
||||||
self._progress.update(1)
|
self._progress.update(1)
|
||||||
if not n.location.valid:
|
if not n.location.valid:
|
||||||
|
|
@ -39,16 +87,28 @@ class PlaceHandler(osmium.SimpleHandler):
|
||||||
lat, lon = n.location.lat, n.location.lon
|
lat, lon = n.location.lat, n.location.lon
|
||||||
if not (UK_BBOX_SOUTH <= lat <= UK_BBOX_NORTH and UK_BBOX_WEST <= lon <= UK_BBOX_EAST):
|
if not (UK_BBOX_SOUTH <= lat <= UK_BBOX_NORTH and UK_BBOX_WEST <= lon <= UK_BBOX_EAST):
|
||||||
return
|
return
|
||||||
place_type = n.tags.get("place")
|
|
||||||
if place_type not in PLACE_TYPES:
|
|
||||||
return
|
|
||||||
name = n.tags.get("name:en", n.tags.get("name", ""))
|
name = n.tags.get("name:en", n.tags.get("name", ""))
|
||||||
if not name:
|
if not name:
|
||||||
return
|
return
|
||||||
self.places.append(
|
|
||||||
{"name": name, "place_type": place_type, "lat": lat, "lon": lon}
|
pop_str = n.tags.get("population", "")
|
||||||
)
|
try:
|
||||||
self._progress.set_postfix(places=f"{len(self.places):,}", refresh=False)
|
population = int(pop_str)
|
||||||
|
except ValueError:
|
||||||
|
population = 0
|
||||||
|
|
||||||
|
# place=* nodes (cities, towns, suburbs, etc.)
|
||||||
|
place_type = n.tags.get("place")
|
||||||
|
if place_type in PLACE_TYPES:
|
||||||
|
self._add(name, place_type, lat, lon, population)
|
||||||
|
return
|
||||||
|
|
||||||
|
# railway=station nodes (tube, national rail, DLR, tram, etc.)
|
||||||
|
if n.tags.get("railway") == "station":
|
||||||
|
display_name = _station_display_name(name, dict(n.tags))
|
||||||
|
self._add(display_name, "station", lat, lon, population)
|
||||||
|
return
|
||||||
|
|
||||||
|
|
||||||
def main() -> None:
|
def main() -> None:
|
||||||
|
|
@ -73,7 +133,7 @@ def main() -> None:
|
||||||
else:
|
else:
|
||||||
print(f"Using cached PBF: {pbf_file}")
|
print(f"Using cached PBF: {pbf_file}")
|
||||||
|
|
||||||
print(f"Extracting place nodes: {sorted(PLACE_TYPES)}")
|
print(f"Extracting place nodes: {sorted(PLACE_TYPES)} + railway=station")
|
||||||
with tqdm(
|
with tqdm(
|
||||||
unit=" elements",
|
unit=" elements",
|
||||||
unit_scale=True,
|
unit_scale=True,
|
||||||
|
|
|
||||||
121
pipeline/transform/_price_utils.py
Normal file
121
pipeline/transform/_price_utils.py
Normal file
|
|
@ -0,0 +1,121 @@
|
||||||
|
"""Shared utilities for price index, price estimate, and renovation premium scripts."""
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import polars as pl
|
||||||
|
|
||||||
|
CURRENT_YEAR = 2025
|
||||||
|
TERRACE_TYPES = [
|
||||||
|
"Mid-Terrace",
|
||||||
|
"End-Terrace",
|
||||||
|
"Enclosed Mid-Terrace",
|
||||||
|
"Enclosed End-Terrace",
|
||||||
|
"Terraced",
|
||||||
|
]
|
||||||
|
FLAT_TYPES = ["Flats/Maisonettes", "Flat", "Maisonette"]
|
||||||
|
TYPE_GROUPS = ["Detached", "Semi-Detached", "Terraced", "Flats", "Bungalow"]
|
||||||
|
SHRINKAGE_K = 50
|
||||||
|
|
||||||
|
|
||||||
|
def type_group_expr():
|
||||||
|
"""Polars expression: Property type -> type_group."""
|
||||||
|
return (
|
||||||
|
pl.when(pl.col("Property type").is_in(TERRACE_TYPES))
|
||||||
|
.then(pl.lit("Terraced"))
|
||||||
|
.when(pl.col("Property type").is_in(FLAT_TYPES))
|
||||||
|
.then(pl.lit("Flats"))
|
||||||
|
.when(pl.col("Property type") == "Bungalow")
|
||||||
|
.then(pl.lit("Bungalow"))
|
||||||
|
.when(pl.col("Property type").is_in(["Detached", "Semi-Detached"]))
|
||||||
|
.then(pl.col("Property type"))
|
||||||
|
.otherwise(pl.lit(None))
|
||||||
|
.alias("type_group")
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def sector_expr():
|
||||||
|
"""Polars expression: Postcode -> sector (drop last 2 chars, strip)."""
|
||||||
|
return (
|
||||||
|
pl.col("Postcode")
|
||||||
|
.str.slice(0, pl.col("Postcode").str.len_chars() - 2)
|
||||||
|
.str.strip_chars()
|
||||||
|
.alias("sector")
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def hierarchy_keys(sector: str) -> tuple[str, str]:
|
||||||
|
"""Return (district, area) for a sector string."""
|
||||||
|
district = sector.rsplit(" ", 1)[0] if " " in sector else sector
|
||||||
|
area = ""
|
||||||
|
for ch in district:
|
||||||
|
if ch.isalpha():
|
||||||
|
area += ch
|
||||||
|
else:
|
||||||
|
break
|
||||||
|
return district, area
|
||||||
|
|
||||||
|
|
||||||
|
AGE_BREAKS = [1900, 1930, 1950, 1967, 1983, 2000, 2010]
|
||||||
|
AGE_LABELS = [
|
||||||
|
"pre-1900",
|
||||||
|
"1900-1929",
|
||||||
|
"1930-1949",
|
||||||
|
"1950-1966",
|
||||||
|
"1967-1982",
|
||||||
|
"1983-1999",
|
||||||
|
"2000-2009",
|
||||||
|
"2010+",
|
||||||
|
]
|
||||||
|
|
||||||
|
HEDONIC_COLUMNS = [
|
||||||
|
"Last known price",
|
||||||
|
"Date of last transaction",
|
||||||
|
"Property type",
|
||||||
|
"Total floor area (sqm)",
|
||||||
|
"Postcode",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def age_band_expr():
|
||||||
|
"""Polars expression: Construction age (UInt16 year) → age band string."""
|
||||||
|
expr = pl.when(pl.col("Construction age").is_null()).then(pl.lit(None))
|
||||||
|
for i, brk in enumerate(AGE_BREAKS):
|
||||||
|
expr = expr.when(pl.col("Construction age") < brk).then(pl.lit(AGE_LABELS[i]))
|
||||||
|
return expr.otherwise(pl.lit(AGE_LABELS[-1])).alias("age_band")
|
||||||
|
|
||||||
|
|
||||||
|
NON_REF_TYPES = ["Terraced", "Semi-Detached", "Flats", "Bungalow"]
|
||||||
|
|
||||||
|
|
||||||
|
def build_hedonic_features(df: pl.DataFrame) -> np.ndarray:
|
||||||
|
"""Build hedonic feature matrix from a DataFrame with type_group column.
|
||||||
|
|
||||||
|
Columns (5 total): log(floor_area), 4 type dummies (ref: Detached).
|
||||||
|
Sector fixed effects do the heavy lifting — additional property features
|
||||||
|
(EPC, rooms, age) add no predictive value after sector demeaning.
|
||||||
|
"""
|
||||||
|
fa = df["Total floor area (sqm)"].to_numpy().astype(np.float32)
|
||||||
|
log_fa = np.log(np.maximum(fa, 1.0)).reshape(-1, 1)
|
||||||
|
tg = df["type_group"].to_numpy()
|
||||||
|
parts = [log_fa]
|
||||||
|
for t in NON_REF_TYPES:
|
||||||
|
parts.append((tg == t).astype(np.float32).reshape(-1, 1))
|
||||||
|
return np.hstack(parts)
|
||||||
|
|
||||||
|
|
||||||
|
def extract_centroids(input_path) -> dict[str, tuple[float, float]]:
|
||||||
|
"""Compute mean lat/lon per postcode sector."""
|
||||||
|
print("Computing sector centroids...")
|
||||||
|
df = (
|
||||||
|
pl.scan_parquet(input_path)
|
||||||
|
.select("Postcode", "lat", "lon")
|
||||||
|
.filter(pl.col("Postcode").is_not_null(), pl.col("lat").is_not_null())
|
||||||
|
.with_columns(sector_expr())
|
||||||
|
.group_by("sector")
|
||||||
|
.agg(pl.col("lat").mean(), pl.col("lon").mean())
|
||||||
|
.collect()
|
||||||
|
)
|
||||||
|
centroids = {}
|
||||||
|
for row in df.iter_rows(named=True):
|
||||||
|
centroids[row["sector"]] = (row["lat"], row["lon"])
|
||||||
|
print(f" {len(centroids):,} sector centroids")
|
||||||
|
return centroids
|
||||||
300
pipeline/transform/hedonic_quality.py
Normal file
300
pipeline/transform/hedonic_quality.py
Normal file
|
|
@ -0,0 +1,300 @@
|
||||||
|
"""Cross-Sectional Hedonic Model (Per-Type)
|
||||||
|
|
||||||
|
Trains separate OLS models per property type on recent sales (last 5 years)
|
||||||
|
with sector fixed effects via Frisch-Waugh-Lovell demeaning:
|
||||||
|
|
||||||
|
log(price) = beta_type * log(floor_area) + alpha_sector_type + epsilon
|
||||||
|
|
||||||
|
Each type gets its own floor area elasticity and sector intercepts, capturing
|
||||||
|
that detached houses (beta=0.74) have higher price sensitivity to size than
|
||||||
|
terraced houses (beta=0.60), and a sector's value differs by property type.
|
||||||
|
|
||||||
|
Sector intercepts are hierarchically shrunk (sector → district → area → national)
|
||||||
|
and spatially smoothed via KD-tree nearest neighbors.
|
||||||
|
|
||||||
|
Output: hedonic_model.json with per-type betas and sector intercepts.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import polars as pl
|
||||||
|
from scipy.spatial import KDTree
|
||||||
|
|
||||||
|
from pipeline.transform._price_utils import (
|
||||||
|
CURRENT_YEAR,
|
||||||
|
HEDONIC_COLUMNS,
|
||||||
|
SHRINKAGE_K,
|
||||||
|
TYPE_GROUPS,
|
||||||
|
extract_centroids,
|
||||||
|
hierarchy_keys,
|
||||||
|
sector_expr,
|
||||||
|
type_group_expr,
|
||||||
|
)
|
||||||
|
|
||||||
|
TRAINING_YEARS = 5
|
||||||
|
SPATIAL_NEIGHBORS = 5
|
||||||
|
SPATIAL_BLEND_K = 30
|
||||||
|
|
||||||
|
|
||||||
|
def load_training_data(input_path: Path) -> pl.DataFrame:
|
||||||
|
"""Load recent sales with complete hedonic features."""
|
||||||
|
min_year = CURRENT_YEAR - TRAINING_YEARS
|
||||||
|
print(f"Loading training data (sales {min_year}-{CURRENT_YEAR})...")
|
||||||
|
df = (
|
||||||
|
pl.scan_parquet(input_path)
|
||||||
|
.select(*HEDONIC_COLUMNS)
|
||||||
|
.filter(
|
||||||
|
pl.col("Last known price").is_not_null(),
|
||||||
|
pl.col("Total floor area (sqm)").is_not_null(),
|
||||||
|
pl.col("Total floor area (sqm)") > 0,
|
||||||
|
pl.col("Postcode").is_not_null(),
|
||||||
|
)
|
||||||
|
.with_columns(
|
||||||
|
pl.col("Date of last transaction").dt.year().alias("sale_year"),
|
||||||
|
type_group_expr(),
|
||||||
|
sector_expr(),
|
||||||
|
)
|
||||||
|
.filter(
|
||||||
|
pl.col("type_group").is_not_null(),
|
||||||
|
pl.col("sale_year").is_not_null(),
|
||||||
|
pl.col("sale_year") >= min_year,
|
||||||
|
pl.col("sale_year") <= CURRENT_YEAR,
|
||||||
|
)
|
||||||
|
.collect()
|
||||||
|
)
|
||||||
|
print(f" {len(df):,} complete cases")
|
||||||
|
return df
|
||||||
|
|
||||||
|
|
||||||
|
def train_type_model(
|
||||||
|
df: pl.DataFrame, type_group: str
|
||||||
|
) -> tuple[float, dict[str, float], dict[str, int], float]:
|
||||||
|
"""Train hedonic model for a single property type.
|
||||||
|
|
||||||
|
Returns (beta_fa, sector_intercepts, sector_counts, national_intercept).
|
||||||
|
"""
|
||||||
|
t_df = df.filter(pl.col("type_group") == type_group)
|
||||||
|
y = np.log(t_df["Last known price"].to_numpy().astype(np.float64))
|
||||||
|
log_fa = np.log(
|
||||||
|
np.maximum(t_df["Total floor area (sqm)"].to_numpy().astype(np.float64), 1.0)
|
||||||
|
)
|
||||||
|
X = log_fa.reshape(-1, 1)
|
||||||
|
sectors = t_df["sector"].to_list()
|
||||||
|
|
||||||
|
# Group by sector for demeaning
|
||||||
|
sector_indices: dict[str, list[int]] = {}
|
||||||
|
for i, s in enumerate(sectors):
|
||||||
|
sector_indices.setdefault(s, []).append(i)
|
||||||
|
|
||||||
|
# Compute sector means and demean
|
||||||
|
X_demeaned = np.empty_like(X)
|
||||||
|
y_demeaned = np.empty_like(y)
|
||||||
|
sector_X_means: dict[str, np.ndarray] = {}
|
||||||
|
sector_y_means: dict[str, float] = {}
|
||||||
|
sector_counts: dict[str, int] = {}
|
||||||
|
|
||||||
|
for s, idxs in sector_indices.items():
|
||||||
|
idx = np.array(idxs)
|
||||||
|
X_mean = X[idx].mean(axis=0)
|
||||||
|
y_mean = y[idx].mean()
|
||||||
|
sector_X_means[s] = X_mean
|
||||||
|
sector_y_means[s] = y_mean
|
||||||
|
X_demeaned[idx] = X[idx] - X_mean
|
||||||
|
y_demeaned[idx] = y[idx] - y_mean
|
||||||
|
sector_counts[s] = len(idxs)
|
||||||
|
|
||||||
|
# OLS on demeaned data
|
||||||
|
beta = np.linalg.lstsq(X_demeaned, y_demeaned, rcond=None)[0]
|
||||||
|
beta_fa = float(beta[0])
|
||||||
|
|
||||||
|
# Recover sector intercepts
|
||||||
|
sector_intercepts = {}
|
||||||
|
for s in sector_indices:
|
||||||
|
sector_intercepts[s] = float(sector_y_means[s] - beta_fa * sector_X_means[s][0])
|
||||||
|
|
||||||
|
national_intercept = float(np.mean(list(sector_intercepts.values())))
|
||||||
|
|
||||||
|
# R-squared
|
||||||
|
y_pred = X[:, 0] * beta_fa
|
||||||
|
for i, s in enumerate(sectors):
|
||||||
|
y_pred[i] += sector_intercepts[s]
|
||||||
|
ss_res = np.sum((y - y_pred) ** 2)
|
||||||
|
ss_tot = np.sum((y - y.mean()) ** 2)
|
||||||
|
r2 = 1 - ss_res / ss_tot
|
||||||
|
|
||||||
|
print(
|
||||||
|
f" {type_group:<15s}: n={len(t_df):>9,} β_fa={beta_fa:.4f} "
|
||||||
|
f"R²={r2:.4f} sectors={len(sector_intercepts):,}"
|
||||||
|
)
|
||||||
|
|
||||||
|
return beta_fa, sector_intercepts, sector_counts, national_intercept
|
||||||
|
|
||||||
|
|
||||||
|
def shrink_intercepts(
|
||||||
|
sector_intercepts: dict[str, float],
|
||||||
|
sector_counts: dict[str, int],
|
||||||
|
) -> dict[str, float]:
|
||||||
|
"""Hierarchical shrinkage: sector -> district -> area -> national."""
|
||||||
|
national = float(np.mean(list(sector_intercepts.values())))
|
||||||
|
|
||||||
|
sector_to_dist: dict[str, str] = {}
|
||||||
|
dist_to_area: dict[str, str] = {}
|
||||||
|
for s in sector_intercepts:
|
||||||
|
d, a = hierarchy_keys(s)
|
||||||
|
sector_to_dist[s] = d
|
||||||
|
dist_to_area[d] = a
|
||||||
|
|
||||||
|
# Area-level intercepts (weighted mean of sectors in area)
|
||||||
|
area_vals: dict[str, list[tuple[float, int]]] = {}
|
||||||
|
for s, val in sector_intercepts.items():
|
||||||
|
d = sector_to_dist[s]
|
||||||
|
a = dist_to_area[d]
|
||||||
|
area_vals.setdefault(a, []).append((val, sector_counts.get(s, 0)))
|
||||||
|
|
||||||
|
area_intercepts: dict[str, float] = {}
|
||||||
|
area_counts: dict[str, int] = {}
|
||||||
|
for a, entries in area_vals.items():
|
||||||
|
total_n = sum(n for _, n in entries)
|
||||||
|
if total_n > 0:
|
||||||
|
area_intercepts[a] = sum(v * n for v, n in entries) / total_n
|
||||||
|
else:
|
||||||
|
area_intercepts[a] = sum(v for v, _ in entries) / len(entries)
|
||||||
|
area_counts[a] = total_n
|
||||||
|
|
||||||
|
# District-level intercepts
|
||||||
|
dist_vals: dict[str, list[tuple[float, int]]] = {}
|
||||||
|
for s, val in sector_intercepts.items():
|
||||||
|
d = sector_to_dist[s]
|
||||||
|
dist_vals.setdefault(d, []).append((val, sector_counts.get(s, 0)))
|
||||||
|
|
||||||
|
dist_intercepts: dict[str, float] = {}
|
||||||
|
dist_counts: dict[str, int] = {}
|
||||||
|
for d, entries in dist_vals.items():
|
||||||
|
total_n = sum(n for _, n in entries)
|
||||||
|
if total_n > 0:
|
||||||
|
dist_intercepts[d] = sum(v * n for v, n in entries) / total_n
|
||||||
|
else:
|
||||||
|
dist_intercepts[d] = sum(v for v, _ in entries) / len(entries)
|
||||||
|
dist_counts[d] = total_n
|
||||||
|
|
||||||
|
# Shrink: area -> national
|
||||||
|
area_shrunk: dict[str, float] = {}
|
||||||
|
for a, val in area_intercepts.items():
|
||||||
|
n = area_counts[a]
|
||||||
|
w = n / (n + SHRINKAGE_K)
|
||||||
|
area_shrunk[a] = w * val + (1 - w) * national
|
||||||
|
|
||||||
|
# Shrink: district -> area
|
||||||
|
dist_shrunk: dict[str, float] = {}
|
||||||
|
for d, val in dist_intercepts.items():
|
||||||
|
a = dist_to_area[d]
|
||||||
|
parent = area_shrunk.get(a, national)
|
||||||
|
n = dist_counts[d]
|
||||||
|
w = n / (n + SHRINKAGE_K)
|
||||||
|
dist_shrunk[d] = w * val + (1 - w) * parent
|
||||||
|
|
||||||
|
# Shrink: sector -> district
|
||||||
|
result: dict[str, float] = {}
|
||||||
|
for s, val in sector_intercepts.items():
|
||||||
|
d = sector_to_dist[s]
|
||||||
|
parent = dist_shrunk.get(d, national)
|
||||||
|
n = sector_counts.get(s, 0)
|
||||||
|
w = n / (n + SHRINKAGE_K)
|
||||||
|
result[s] = w * val + (1 - w) * parent
|
||||||
|
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def spatial_smooth_intercepts(
|
||||||
|
sector_intercepts: dict[str, float],
|
||||||
|
centroids: dict[str, tuple[float, float]],
|
||||||
|
sector_counts: dict[str, int],
|
||||||
|
) -> dict[str, float]:
|
||||||
|
"""Blend sparse sector intercepts with K nearest neighbors."""
|
||||||
|
sectors_with_coords = [s for s in sector_intercepts if s in centroids]
|
||||||
|
if len(sectors_with_coords) < SPATIAL_NEIGHBORS + 1:
|
||||||
|
return sector_intercepts
|
||||||
|
|
||||||
|
coords = np.array([centroids[s] for s in sectors_with_coords])
|
||||||
|
mean_lat = np.mean(coords[:, 0])
|
||||||
|
scale = np.cos(np.radians(mean_lat))
|
||||||
|
scaled_coords = np.column_stack([coords[:, 0], coords[:, 1] * scale])
|
||||||
|
tree = KDTree(scaled_coords)
|
||||||
|
|
||||||
|
result = dict(sector_intercepts)
|
||||||
|
for i, sec in enumerate(sectors_with_coords):
|
||||||
|
n = sector_counts.get(sec, 0)
|
||||||
|
self_w = n / (n + SPATIAL_BLEND_K)
|
||||||
|
if self_w > 0.95:
|
||||||
|
continue
|
||||||
|
|
||||||
|
dists, idxs = tree.query(scaled_coords[i], k=SPATIAL_NEIGHBORS + 1)
|
||||||
|
neighbor_dists = dists[1:]
|
||||||
|
neighbor_idxs = idxs[1:]
|
||||||
|
|
||||||
|
inv_dists = []
|
||||||
|
neighbor_vals = []
|
||||||
|
for d, j in zip(neighbor_dists, neighbor_idxs):
|
||||||
|
ns = sectors_with_coords[j]
|
||||||
|
if d > 0 and ns in sector_intercepts:
|
||||||
|
inv_dists.append(1.0 / d)
|
||||||
|
neighbor_vals.append(sector_intercepts[ns])
|
||||||
|
|
||||||
|
if not neighbor_vals:
|
||||||
|
continue
|
||||||
|
|
||||||
|
total_inv = sum(inv_dists)
|
||||||
|
nbr_w = 1.0 - self_w
|
||||||
|
blended = self_w * sector_intercepts[sec]
|
||||||
|
for val, iw in zip(neighbor_vals, inv_dists):
|
||||||
|
blended += nbr_w * (iw / total_inv) * val
|
||||||
|
result[sec] = blended
|
||||||
|
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
parser = argparse.ArgumentParser(description="Train cross-sectional hedonic model")
|
||||||
|
parser.add_argument(
|
||||||
|
"--input", type=Path, required=True, help="Path to wide.parquet"
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--output", type=Path, required=True, help="Output hedonic_model.json"
|
||||||
|
)
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
df = load_training_data(args.input)
|
||||||
|
centroids = extract_centroids(args.input)
|
||||||
|
|
||||||
|
print("\nTraining per-type models...")
|
||||||
|
type_models = {}
|
||||||
|
total_sectors = 0
|
||||||
|
|
||||||
|
for tg in TYPE_GROUPS:
|
||||||
|
beta_fa, raw_intercepts, sector_counts, national = train_type_model(df, tg)
|
||||||
|
|
||||||
|
shrunk = shrink_intercepts(raw_intercepts, sector_counts)
|
||||||
|
smoothed = spatial_smooth_intercepts(shrunk, centroids, sector_counts)
|
||||||
|
total_sectors += len(smoothed)
|
||||||
|
|
||||||
|
type_models[tg] = {
|
||||||
|
"beta_fa": beta_fa,
|
||||||
|
"sector_intercepts": smoothed,
|
||||||
|
"national_intercept": national,
|
||||||
|
}
|
||||||
|
|
||||||
|
# Output
|
||||||
|
args.output.parent.mkdir(parents=True, exist_ok=True)
|
||||||
|
with open(args.output, "w") as f:
|
||||||
|
json.dump({"type_models": type_models}, f, indent=2)
|
||||||
|
|
||||||
|
size_kb = args.output.stat().st_size / 1024
|
||||||
|
print(f"\nWrote {args.output} ({size_kb:.0f} KB)")
|
||||||
|
print(f" {len(TYPE_GROUPS)} type models, {total_sectors:,} total sector intercepts")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
|
|
@ -223,7 +223,6 @@ def _build_wide(
|
||||||
)
|
)
|
||||||
.drop(
|
.drop(
|
||||||
"inspection_date",
|
"inspection_date",
|
||||||
"floor_height",
|
|
||||||
"_bedrooms",
|
"_bedrooms",
|
||||||
"LSOA name (2021)",
|
"LSOA name (2021)",
|
||||||
"Local Authority District code (2024)",
|
"Local Authority District code (2024)",
|
||||||
|
|
@ -276,6 +275,7 @@ def _build_wide(
|
||||||
"shrink_swell_risk": "Shrink-swell risk",
|
"shrink_swell_risk": "Shrink-swell risk",
|
||||||
"soluble_rocks_risk": "Soluble rocks risk",
|
"soluble_rocks_risk": "Soluble rocks risk",
|
||||||
"median_monthly_rent": "Estimated monthly rent",
|
"median_monthly_rent": "Estimated monthly rent",
|
||||||
|
"floor_height": "Interior height (m)",
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
)
|
)
|
||||||
|
|
|
||||||
|
|
@ -9,45 +9,60 @@ Output: backtest_results.parquet with predictions vs actuals.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
|
import json
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import numpy as np
|
import numpy as np
|
||||||
import polars as pl
|
import polars as pl
|
||||||
|
|
||||||
CURRENT_YEAR = 2025
|
from pipeline.transform._price_utils import (
|
||||||
|
CURRENT_YEAR,
|
||||||
|
HEDONIC_COLUMNS,
|
||||||
|
sector_expr,
|
||||||
|
type_group_expr,
|
||||||
|
)
|
||||||
|
|
||||||
TEST_YEAR_MIN = 2022
|
TEST_YEAR_MIN = 2022
|
||||||
TERRACE_TYPES = ["Mid-Terrace", "End-Terrace", "Enclosed Mid-Terrace", "Enclosed End-Terrace"]
|
|
||||||
|
|
||||||
|
|
||||||
def type_group_expr():
|
def extract_test_set(
|
||||||
return (
|
input_path: Path, include_hedonic_cols: bool = False
|
||||||
pl.when(pl.col("Property type").is_in(TERRACE_TYPES)).then(pl.lit("Terraced"))
|
) -> pl.DataFrame:
|
||||||
.when(pl.col("Property type") == "Flats/Maisonettes").then(pl.lit("Flats"))
|
|
||||||
.when(pl.col("Property type").is_in(["Detached", "Semi-Detached"])).then(pl.col("Property type"))
|
|
||||||
.otherwise(pl.lit(None))
|
|
||||||
.alias("type_group")
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def extract_test_set(input_path: Path) -> pl.DataFrame:
|
|
||||||
"""Extract test pairs: second-to-last sale as input, last sale as ground truth."""
|
"""Extract test pairs: second-to-last sale as input, last sale as ground truth."""
|
||||||
print("Loading test set...")
|
print("Loading test set...")
|
||||||
|
cols = ["Postcode", "historical_prices", "Property type"]
|
||||||
|
if include_hedonic_cols:
|
||||||
|
for c in HEDONIC_COLUMNS:
|
||||||
|
if c not in cols:
|
||||||
|
cols.append(c)
|
||||||
df = (
|
df = (
|
||||||
pl.scan_parquet(input_path)
|
pl.scan_parquet(input_path)
|
||||||
.select("Postcode", "historical_prices", "Property type")
|
.select(cols)
|
||||||
.filter(
|
.filter(
|
||||||
pl.col("Postcode").is_not_null(),
|
pl.col("Postcode").is_not_null(),
|
||||||
pl.col("historical_prices").list.len() >= 2,
|
pl.col("historical_prices").list.len() >= 2,
|
||||||
)
|
)
|
||||||
.with_columns(
|
.with_columns(
|
||||||
pl.col("Postcode").str.slice(0, pl.col("Postcode").str.len_chars() - 2).str.strip_chars().alias("sector"),
|
sector_expr(),
|
||||||
type_group_expr(),
|
type_group_expr(),
|
||||||
# Last sale (ground truth)
|
# Last sale (ground truth)
|
||||||
pl.col("historical_prices").list.last().struct.field("year").alias("actual_year"),
|
pl.col("historical_prices")
|
||||||
pl.col("historical_prices").list.last().struct.field("price").alias("actual_price"),
|
.list.last()
|
||||||
|
.struct.field("year")
|
||||||
|
.alias("actual_year"),
|
||||||
|
pl.col("historical_prices")
|
||||||
|
.list.last()
|
||||||
|
.struct.field("price")
|
||||||
|
.alias("actual_price"),
|
||||||
# Second-to-last sale (input)
|
# Second-to-last sale (input)
|
||||||
pl.col("historical_prices").list.get(-2).struct.field("year").alias("input_year"),
|
pl.col("historical_prices")
|
||||||
pl.col("historical_prices").list.get(-2).struct.field("price").alias("input_price"),
|
.list.get(-2)
|
||||||
|
.struct.field("year")
|
||||||
|
.alias("input_year"),
|
||||||
|
pl.col("historical_prices")
|
||||||
|
.list.get(-2)
|
||||||
|
.struct.field("price")
|
||||||
|
.alias("input_price"),
|
||||||
)
|
)
|
||||||
.filter(
|
.filter(
|
||||||
pl.col("actual_year") >= TEST_YEAR_MIN,
|
pl.col("actual_year") >= TEST_YEAR_MIN,
|
||||||
|
|
@ -71,7 +86,9 @@ def predict(test: pl.DataFrame, index: pl.DataFrame) -> pl.DataFrame:
|
||||||
|
|
||||||
# Join type-specific index at input year
|
# Join type-specific index at input year
|
||||||
test = test.join(
|
test = test.join(
|
||||||
idx_typed.select("sector", "type_group", "year", pl.col("log_index").alias("li_in_typed")),
|
idx_typed.select(
|
||||||
|
"sector", "type_group", "year", pl.col("log_index").alias("li_in_typed")
|
||||||
|
),
|
||||||
left_on=["sector", "type_group", "input_year"],
|
left_on=["sector", "type_group", "input_year"],
|
||||||
right_on=["sector", "type_group", "year"],
|
right_on=["sector", "type_group", "year"],
|
||||||
how="left",
|
how="left",
|
||||||
|
|
@ -85,7 +102,12 @@ def predict(test: pl.DataFrame, index: pl.DataFrame) -> pl.DataFrame:
|
||||||
)
|
)
|
||||||
# Join type-specific index at actual year
|
# Join type-specific index at actual year
|
||||||
test = test.join(
|
test = test.join(
|
||||||
idx_typed.select("sector", "type_group", "year", pl.col("log_index").alias("li_act_typed")),
|
idx_typed.select(
|
||||||
|
"sector",
|
||||||
|
"type_group",
|
||||||
|
"year",
|
||||||
|
pl.col("log_index").alias("li_act_typed"),
|
||||||
|
),
|
||||||
left_on=["sector", "type_group", "actual_year"],
|
left_on=["sector", "type_group", "actual_year"],
|
||||||
right_on=["sector", "type_group", "year"],
|
right_on=["sector", "type_group", "year"],
|
||||||
how="left",
|
how="left",
|
||||||
|
|
@ -99,19 +121,27 @@ def predict(test: pl.DataFrame, index: pl.DataFrame) -> pl.DataFrame:
|
||||||
)
|
)
|
||||||
|
|
||||||
test = test.with_columns(
|
test = test.with_columns(
|
||||||
pl.col("li_in_typed").fill_null(pl.col("li_in_all")).alias("log_index_input"),
|
pl.col("li_in_typed")
|
||||||
pl.col("li_act_typed").fill_null(pl.col("li_act_all")).alias("log_index_actual"),
|
.fill_null(pl.col("li_in_all"))
|
||||||
|
.alias("log_index_input"),
|
||||||
|
pl.col("li_act_typed")
|
||||||
|
.fill_null(pl.col("li_act_all"))
|
||||||
|
.alias("log_index_actual"),
|
||||||
)
|
)
|
||||||
else:
|
else:
|
||||||
# Unstratified index
|
# Unstratified index
|
||||||
test = test.join(
|
test = test.join(
|
||||||
index.select("sector", "year", pl.col("log_index").alias("log_index_input")),
|
index.select(
|
||||||
|
"sector", "year", pl.col("log_index").alias("log_index_input")
|
||||||
|
),
|
||||||
left_on=["sector", "input_year"],
|
left_on=["sector", "input_year"],
|
||||||
right_on=["sector", "year"],
|
right_on=["sector", "year"],
|
||||||
how="left",
|
how="left",
|
||||||
)
|
)
|
||||||
test = test.join(
|
test = test.join(
|
||||||
index.select("sector", "year", pl.col("log_index").alias("log_index_actual")),
|
index.select(
|
||||||
|
"sector", "year", pl.col("log_index").alias("log_index_actual")
|
||||||
|
),
|
||||||
left_on=["sector", "actual_year"],
|
left_on=["sector", "actual_year"],
|
||||||
right_on=["sector", "year"],
|
right_on=["sector", "year"],
|
||||||
how="left",
|
how="left",
|
||||||
|
|
@ -121,7 +151,9 @@ def predict(test: pl.DataFrame, index: pl.DataFrame) -> pl.DataFrame:
|
||||||
(
|
(
|
||||||
pl.col("input_price").cast(pl.Float64)
|
pl.col("input_price").cast(pl.Float64)
|
||||||
* (pl.col("log_index_actual") - pl.col("log_index_input")).exp()
|
* (pl.col("log_index_actual") - pl.col("log_index_input")).exp()
|
||||||
).fill_null(pl.col("input_price").cast(pl.Float64)).alias("predicted"),
|
)
|
||||||
|
.fill_null(pl.col("input_price").cast(pl.Float64))
|
||||||
|
.alias("predicted"),
|
||||||
)
|
)
|
||||||
return test
|
return test
|
||||||
|
|
||||||
|
|
@ -150,7 +182,15 @@ def print_metrics_table(metrics_by_stage: dict):
|
||||||
print("BACKTEST RESULTS")
|
print("BACKTEST RESULTS")
|
||||||
print("=" * 55)
|
print("=" * 55)
|
||||||
|
|
||||||
metric_names = ["MdAPE (%)", "% within 10%", "% within 20%", "% within 30%", "MAE (£)", "Mean signed error (£)", "n"]
|
metric_names = [
|
||||||
|
"MdAPE (%)",
|
||||||
|
"% within 10%",
|
||||||
|
"% within 20%",
|
||||||
|
"% within 30%",
|
||||||
|
"MAE (£)",
|
||||||
|
"Mean signed error (£)",
|
||||||
|
"n",
|
||||||
|
]
|
||||||
stages = list(metrics_by_stage.keys())
|
stages = list(metrics_by_stage.keys())
|
||||||
|
|
||||||
header = f"{'Metric':<25s}"
|
header = f"{'Metric':<25s}"
|
||||||
|
|
@ -176,20 +216,37 @@ def print_metrics_table(metrics_by_stage: dict):
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
parser = argparse.ArgumentParser(description="Backtest price estimation model")
|
parser = argparse.ArgumentParser(description="Backtest price estimation model")
|
||||||
parser.add_argument("--input", type=Path, required=True, help="Path to wide.parquet")
|
parser.add_argument(
|
||||||
parser.add_argument("--index", type=Path, required=True, help="Path to price_index.parquet")
|
"--input", type=Path, required=True, help="Path to wide.parquet"
|
||||||
parser.add_argument("--output", type=Path, required=True, help="Output backtest_results.parquet")
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--index", type=Path, required=True, help="Path to price_index.parquet"
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--output", type=Path, required=True, help="Output backtest_results.parquet"
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--hedonic-model",
|
||||||
|
type=Path,
|
||||||
|
default=None,
|
||||||
|
help="Path to hedonic_model.json (optional)",
|
||||||
|
)
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
|
|
||||||
index = pl.read_parquet(args.index)
|
index = pl.read_parquet(args.index)
|
||||||
has_type_group = "type_group" in index.columns
|
has_type_group = "type_group" in index.columns
|
||||||
if has_type_group:
|
if has_type_group:
|
||||||
print(f"Price index: {len(index):,} rows, {index['sector'].n_unique():,} sectors, "
|
print(
|
||||||
f"{index['type_group'].n_unique()} type groups")
|
f"Price index: {len(index):,} rows, {index['sector'].n_unique():,} sectors, "
|
||||||
|
f"{index['type_group'].n_unique()} type groups"
|
||||||
|
)
|
||||||
else:
|
else:
|
||||||
print(f"Price index: {len(index):,} rows, {index['sector'].n_unique():,} sectors")
|
print(
|
||||||
|
f"Price index: {len(index):,} rows, {index['sector'].n_unique():,} sectors"
|
||||||
|
)
|
||||||
|
|
||||||
test = extract_test_set(args.input)
|
has_hedonic = args.hedonic_model is not None
|
||||||
|
test = extract_test_set(args.input, include_hedonic_cols=has_hedonic)
|
||||||
|
|
||||||
print("\nPredicting with price index...")
|
print("\nPredicting with price index...")
|
||||||
test = predict(test, index)
|
test = predict(test, index)
|
||||||
|
|
@ -197,19 +254,126 @@ def main():
|
||||||
# Compute and print metrics
|
# Compute and print metrics
|
||||||
actual = test["actual_price"].to_numpy().astype(np.float64)
|
actual = test["actual_price"].to_numpy().astype(np.float64)
|
||||||
metrics = {
|
metrics = {
|
||||||
"Naive": compute_metrics(actual, test["input_price"].to_numpy().astype(np.float64)),
|
"Naive": compute_metrics(
|
||||||
"Index": compute_metrics(actual, test["predicted"].to_numpy().astype(np.float64)),
|
actual, test["input_price"].to_numpy().astype(np.float64)
|
||||||
|
),
|
||||||
|
"Index": compute_metrics(
|
||||||
|
actual, test["predicted"].to_numpy().astype(np.float64)
|
||||||
|
),
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Hedonic blending
|
||||||
|
if has_hedonic:
|
||||||
|
print("\nApplying hedonic blending...")
|
||||||
|
with open(args.hedonic_model) as f:
|
||||||
|
model = json.load(f)
|
||||||
|
type_models = model["type_models"]
|
||||||
|
|
||||||
|
# Identify eligible rows for hedonic estimate
|
||||||
|
hedonic_mask = (
|
||||||
|
pl.col("Total floor area (sqm)").is_not_null()
|
||||||
|
& (pl.col("Total floor area (sqm)") > 0)
|
||||||
|
& pl.col("type_group").is_not_null()
|
||||||
|
)
|
||||||
|
eligible_mask = test.select(hedonic_mask).to_series()
|
||||||
|
eligible = test.filter(eligible_mask)
|
||||||
|
|
||||||
|
if len(eligible) > 0:
|
||||||
|
log_fa = np.log(
|
||||||
|
np.maximum(
|
||||||
|
eligible["Total floor area (sqm)"].to_numpy().astype(np.float64),
|
||||||
|
1.0,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
sectors = eligible["sector"].to_list()
|
||||||
|
types = eligible["type_group"].to_list()
|
||||||
|
|
||||||
|
# Per-type hedonic prediction
|
||||||
|
log_hedonic = np.empty(len(eligible))
|
||||||
|
for i in range(len(eligible)):
|
||||||
|
tm = type_models.get(types[i])
|
||||||
|
if tm is None:
|
||||||
|
log_hedonic[i] = np.nan
|
||||||
|
continue
|
||||||
|
alpha = tm["sector_intercepts"].get(
|
||||||
|
sectors[i], tm["national_intercept"]
|
||||||
|
)
|
||||||
|
log_hedonic[i] = tm["beta_fa"] * log_fa[i] + alpha
|
||||||
|
|
||||||
|
valid = np.isfinite(log_hedonic)
|
||||||
|
|
||||||
|
# Hold years: input_year to actual_year (simulating real prediction)
|
||||||
|
input_years = eligible["input_year"].to_numpy().astype(np.float64)
|
||||||
|
actual_years = eligible["actual_year"].to_numpy().astype(np.float64)
|
||||||
|
hold_years = np.maximum(actual_years - input_years, 0.0)
|
||||||
|
|
||||||
|
log_index_pred = np.log(
|
||||||
|
np.maximum(eligible["predicted"].to_numpy().astype(np.float64), 1.0)
|
||||||
|
)
|
||||||
|
|
||||||
|
# Sweep tau values (only on valid hedonic rows)
|
||||||
|
tau_values = [5.0, 10.0, 15.0, 20.0, 30.0]
|
||||||
|
actual_eligible = eligible["actual_price"].to_numpy().astype(np.float64)
|
||||||
|
best_tau = 15.0
|
||||||
|
best_mdape = float("inf")
|
||||||
|
|
||||||
|
print(f"\n tau sweep ({valid.sum():,} eligible properties):")
|
||||||
|
for tau in tau_values:
|
||||||
|
blend_w = hold_years / (hold_years + tau)
|
||||||
|
log_blended = np.where(
|
||||||
|
valid,
|
||||||
|
(1 - blend_w) * log_index_pred + blend_w * log_hedonic,
|
||||||
|
log_index_pred,
|
||||||
|
)
|
||||||
|
blended = np.exp(log_blended)
|
||||||
|
m = compute_metrics(actual_eligible, blended)
|
||||||
|
marker = ""
|
||||||
|
if m["MdAPE (%)"] < best_mdape:
|
||||||
|
best_mdape = m["MdAPE (%)"]
|
||||||
|
best_tau = tau
|
||||||
|
marker = " <-- best"
|
||||||
|
print(
|
||||||
|
f" tau={tau:>4.0f}: MdAPE={m['MdAPE (%)']:>5.1f}%, "
|
||||||
|
f"within 10%={m['% within 10%']:>5.1f}%{marker}"
|
||||||
|
)
|
||||||
|
|
||||||
|
print(f"\n Best tau = {best_tau}")
|
||||||
|
|
||||||
|
# Compute blended predictions with best tau for full test set
|
||||||
|
blend_w = hold_years / (hold_years + best_tau)
|
||||||
|
log_blended = np.where(
|
||||||
|
valid,
|
||||||
|
(1 - blend_w) * log_index_pred + blend_w * log_hedonic,
|
||||||
|
log_index_pred,
|
||||||
|
)
|
||||||
|
blended_eligible = np.exp(log_blended)
|
||||||
|
|
||||||
|
# Merge back: for non-eligible rows, use index prediction
|
||||||
|
blended_all = test["predicted"].to_numpy().astype(np.float64).copy()
|
||||||
|
eligible_indices = eligible_mask.arg_true()
|
||||||
|
for i, idx in enumerate(eligible_indices):
|
||||||
|
blended_all[idx] = blended_eligible[i]
|
||||||
|
|
||||||
|
test = test.with_columns(
|
||||||
|
pl.Series("blended", blended_all, dtype=pl.Float64),
|
||||||
|
)
|
||||||
|
metrics["Blended"] = compute_metrics(actual, blended_all)
|
||||||
|
|
||||||
print_metrics_table(metrics)
|
print_metrics_table(metrics)
|
||||||
|
|
||||||
# Save results
|
# Save results
|
||||||
result = test.select(
|
result_cols = [
|
||||||
"Postcode", "sector",
|
"Postcode",
|
||||||
"input_year", "input_price",
|
"sector",
|
||||||
"actual_year", "actual_price",
|
"input_year",
|
||||||
|
"input_price",
|
||||||
|
"actual_year",
|
||||||
|
"actual_price",
|
||||||
"predicted",
|
"predicted",
|
||||||
)
|
]
|
||||||
|
if "blended" in test.columns:
|
||||||
|
result_cols.append("blended")
|
||||||
|
result = test.select(result_cols)
|
||||||
|
|
||||||
result.write_parquet(args.output)
|
result.write_parquet(args.output)
|
||||||
size_mb = args.output.stat().st_size / (1024 * 1024)
|
size_mb = args.output.stat().st_size / (1024 * 1024)
|
||||||
|
|
|
||||||
|
|
@ -4,32 +4,56 @@ Joins the precomputed repeat-sales price index (from price_index.py) with each
|
||||||
property's last known sale to produce an inflation-adjusted current price estimate.
|
property's last known sale to produce an inflation-adjusted current price estimate.
|
||||||
Uses type-stratified index when available, falling back to "All" type.
|
Uses type-stratified index when available, falling back to "All" type.
|
||||||
|
|
||||||
|
Optionally applies renovation premiums from renovation_premium.py: for properties
|
||||||
|
with post-sale renovation events, the estimated price is adjusted upward based on
|
||||||
|
data-driven per-area premiums with time decay.
|
||||||
|
|
||||||
Modifies wide.parquet in-place, adding the "Estimated current price" column.
|
Modifies wide.parquet in-place, adding the "Estimated current price" column.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import argparse
|
import argparse
|
||||||
|
import json
|
||||||
|
import math
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
import polars as pl
|
import polars as pl
|
||||||
|
|
||||||
CURRENT_YEAR = 2025
|
from pipeline.transform._price_utils import (
|
||||||
TERRACE_TYPES = ["Mid-Terrace", "End-Terrace", "Enclosed Mid-Terrace", "Enclosed End-Terrace"]
|
CURRENT_YEAR,
|
||||||
|
sector_expr,
|
||||||
|
type_group_expr,
|
||||||
|
)
|
||||||
|
|
||||||
|
HALF_LIFE = 10.0
|
||||||
def type_group_expr():
|
DECAY_RATE = math.log(2) / HALF_LIFE
|
||||||
return (
|
|
||||||
pl.when(pl.col("Property type").is_in(TERRACE_TYPES)).then(pl.lit("Terraced"))
|
|
||||||
.when(pl.col("Property type") == "Flats/Maisonettes").then(pl.lit("Flats"))
|
|
||||||
.when(pl.col("Property type").is_in(["Detached", "Semi-Detached"])).then(pl.col("Property type"))
|
|
||||||
.otherwise(pl.lit(None))
|
|
||||||
.alias("type_group")
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
parser = argparse.ArgumentParser(description="Augment wide.parquet with estimated current prices")
|
parser = argparse.ArgumentParser(
|
||||||
parser.add_argument("--input", type=Path, required=True, help="Path to wide.parquet (modified in-place)")
|
description="Augment wide.parquet with estimated current prices"
|
||||||
parser.add_argument("--index", type=Path, required=True, help="Path to price_index.parquet")
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--input",
|
||||||
|
type=Path,
|
||||||
|
required=True,
|
||||||
|
help="Path to wide.parquet (modified in-place)",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--index", type=Path, required=True, help="Path to price_index.parquet"
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--renovation-premium",
|
||||||
|
type=Path,
|
||||||
|
default=None,
|
||||||
|
help="Path to renovation_premium.parquet (optional)",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--hedonic-model",
|
||||||
|
type=Path,
|
||||||
|
default=None,
|
||||||
|
help="Path to hedonic_model.json (optional)",
|
||||||
|
)
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
|
|
||||||
print("Loading wide.parquet...")
|
print("Loading wide.parquet...")
|
||||||
|
|
@ -49,7 +73,7 @@ def main():
|
||||||
)
|
)
|
||||||
|
|
||||||
df = df.with_columns(
|
df = df.with_columns(
|
||||||
pl.col("Postcode").str.slice(0, pl.col("Postcode").str.len_chars() - 2).str.strip_chars().alias("_sector"),
|
sector_expr().alias("_sector"),
|
||||||
pl.col("Date of last transaction").dt.year().alias("_sale_year"),
|
pl.col("Date of last transaction").dt.year().alias("_sale_year"),
|
||||||
type_group_expr().alias("_type_group"),
|
type_group_expr().alias("_type_group"),
|
||||||
)
|
)
|
||||||
|
|
@ -57,10 +81,14 @@ def main():
|
||||||
index = pl.read_parquet(args.index)
|
index = pl.read_parquet(args.index)
|
||||||
has_type_group = "type_group" in index.columns
|
has_type_group = "type_group" in index.columns
|
||||||
if has_type_group:
|
if has_type_group:
|
||||||
print(f" Price index: {len(index):,} rows, {index['sector'].n_unique():,} sectors, "
|
print(
|
||||||
f"{index['type_group'].n_unique()} type groups")
|
f" Price index: {len(index):,} rows, {index['sector'].n_unique():,} sectors, "
|
||||||
|
f"{index['type_group'].n_unique()} type groups"
|
||||||
|
)
|
||||||
else:
|
else:
|
||||||
print(f" Price index: {len(index):,} rows, {index['sector'].n_unique():,} sectors (unstratified)")
|
print(
|
||||||
|
f" Price index: {len(index):,} rows, {index['sector'].n_unique():,} sectors (unstratified)"
|
||||||
|
)
|
||||||
|
|
||||||
print("\nApplying repeat-sales index...")
|
print("\nApplying repeat-sales index...")
|
||||||
|
|
||||||
|
|
@ -70,49 +98,63 @@ def main():
|
||||||
|
|
||||||
# Join type-specific index at sale year
|
# Join type-specific index at sale year
|
||||||
df = df.join(
|
df = df.join(
|
||||||
idx_typed.select("sector", "type_group", "year", pl.col("log_index").alias("log_idx_sale_typed")),
|
idx_typed.select(
|
||||||
|
"sector",
|
||||||
|
"type_group",
|
||||||
|
"year",
|
||||||
|
pl.col("log_index").alias("log_idx_sale_typed"),
|
||||||
|
),
|
||||||
left_on=["_sector", "_type_group", "_sale_year"],
|
left_on=["_sector", "_type_group", "_sale_year"],
|
||||||
right_on=["sector", "type_group", "year"],
|
right_on=["sector", "type_group", "year"],
|
||||||
how="left",
|
how="left",
|
||||||
)
|
)
|
||||||
# Join "All" index at sale year
|
# Join "All" index at sale year
|
||||||
df = df.join(
|
df = df.join(
|
||||||
idx_all.select("sector", "year", pl.col("log_index").alias("log_idx_sale_all")),
|
idx_all.select(
|
||||||
|
"sector", "year", pl.col("log_index").alias("log_idx_sale_all")
|
||||||
|
),
|
||||||
left_on=["_sector", "_sale_year"],
|
left_on=["_sector", "_sale_year"],
|
||||||
right_on=["sector", "year"],
|
right_on=["sector", "year"],
|
||||||
how="left",
|
how="left",
|
||||||
)
|
)
|
||||||
# Join type-specific index at current year
|
# Join type-specific index at current year
|
||||||
df = df.join(
|
df = df.join(
|
||||||
idx_typed.filter(pl.col("year") == CURRENT_YEAR)
|
idx_typed.filter(pl.col("year") == CURRENT_YEAR).select(
|
||||||
.select("sector", "type_group", pl.col("log_index").alias("log_idx_cur_typed")),
|
"sector", "type_group", pl.col("log_index").alias("log_idx_cur_typed")
|
||||||
|
),
|
||||||
left_on=["_sector", "_type_group"],
|
left_on=["_sector", "_type_group"],
|
||||||
right_on=["sector", "type_group"],
|
right_on=["sector", "type_group"],
|
||||||
how="left",
|
how="left",
|
||||||
)
|
)
|
||||||
# Join "All" index at current year
|
# Join "All" index at current year
|
||||||
df = df.join(
|
df = df.join(
|
||||||
idx_all.filter(pl.col("year") == CURRENT_YEAR)
|
idx_all.filter(pl.col("year") == CURRENT_YEAR).select(
|
||||||
.select("sector", pl.col("log_index").alias("log_idx_cur_all")),
|
"sector", pl.col("log_index").alias("log_idx_cur_all")
|
||||||
|
),
|
||||||
left_on="_sector",
|
left_on="_sector",
|
||||||
right_on="sector",
|
right_on="sector",
|
||||||
how="left",
|
how="left",
|
||||||
)
|
)
|
||||||
|
|
||||||
df = df.with_columns(
|
df = df.with_columns(
|
||||||
pl.col("log_idx_sale_typed").fill_null(pl.col("log_idx_sale_all")).alias("_log_index_sale"),
|
pl.col("log_idx_sale_typed")
|
||||||
pl.col("log_idx_cur_typed").fill_null(pl.col("log_idx_cur_all")).alias("_log_index_current"),
|
.fill_null(pl.col("log_idx_sale_all"))
|
||||||
|
.alias("_log_index_sale"),
|
||||||
|
pl.col("log_idx_cur_typed")
|
||||||
|
.fill_null(pl.col("log_idx_cur_all"))
|
||||||
|
.alias("_log_index_current"),
|
||||||
)
|
)
|
||||||
else:
|
else:
|
||||||
df = df.join(
|
df = df.join(
|
||||||
index.select("sector", "year", pl.col("log_index").alias("_log_index_sale")),
|
index.select(
|
||||||
|
"sector", "year", pl.col("log_index").alias("_log_index_sale")
|
||||||
|
),
|
||||||
left_on=["_sector", "_sale_year"],
|
left_on=["_sector", "_sale_year"],
|
||||||
right_on=["sector", "year"],
|
right_on=["sector", "year"],
|
||||||
how="left",
|
how="left",
|
||||||
)
|
)
|
||||||
index_current = (
|
index_current = index.filter(pl.col("year") == CURRENT_YEAR).select(
|
||||||
index.filter(pl.col("year") == CURRENT_YEAR)
|
"sector", pl.col("log_index").alias("_log_index_current")
|
||||||
.select("sector", pl.col("log_index").alias("_log_index_current"))
|
|
||||||
)
|
)
|
||||||
df = df.join(index_current, left_on="_sector", right_on="sector", how="left")
|
df = df.join(index_current, left_on="_sector", right_on="sector", how="left")
|
||||||
|
|
||||||
|
|
@ -127,6 +169,224 @@ def main():
|
||||||
.alias("Estimated current price"),
|
.alias("Estimated current price"),
|
||||||
)
|
)
|
||||||
|
|
||||||
|
n_adjusted = df.filter(has_price & pl.col("_log_index_sale").is_not_null()).height
|
||||||
|
n_with_price = df.filter(has_price).height
|
||||||
|
print(
|
||||||
|
f" {n_adjusted:,} of {n_with_price:,} properties adjusted by index ({n_adjusted / max(n_with_price, 1) * 100:.1f}%)"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Apply hedonic blending if model provided
|
||||||
|
if args.hedonic_model is not None:
|
||||||
|
print("\nApplying hedonic blending...")
|
||||||
|
with open(args.hedonic_model) as f:
|
||||||
|
model = json.load(f)
|
||||||
|
type_models = model["type_models"]
|
||||||
|
tau = model.get("tau", 15.0)
|
||||||
|
print(f" tau = {tau}, {len(type_models)} type models")
|
||||||
|
|
||||||
|
# Add type_group for per-type lookup
|
||||||
|
df = df.with_columns(type_group_expr())
|
||||||
|
hedonic_mask = (
|
||||||
|
has_price
|
||||||
|
& pl.col("Estimated current price").is_not_null()
|
||||||
|
& pl.col("Total floor area (sqm)").is_not_null()
|
||||||
|
& (pl.col("Total floor area (sqm)") > 0)
|
||||||
|
& pl.col("type_group").is_not_null()
|
||||||
|
)
|
||||||
|
eligible = df.filter(hedonic_mask)
|
||||||
|
|
||||||
|
if len(eligible) > 0:
|
||||||
|
log_fa = np.log(
|
||||||
|
np.maximum(
|
||||||
|
eligible["Total floor area (sqm)"].to_numpy().astype(np.float64),
|
||||||
|
1.0,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
sectors = eligible["_sector"].to_list()
|
||||||
|
types = eligible["type_group"].to_list()
|
||||||
|
|
||||||
|
# Per-type hedonic prediction
|
||||||
|
log_hedonic = np.empty(len(eligible))
|
||||||
|
for i in range(len(eligible)):
|
||||||
|
tm = type_models.get(types[i])
|
||||||
|
if tm is None:
|
||||||
|
log_hedonic[i] = np.nan
|
||||||
|
continue
|
||||||
|
alpha = tm["sector_intercepts"].get(
|
||||||
|
sectors[i], tm["national_intercept"]
|
||||||
|
)
|
||||||
|
log_hedonic[i] = tm["beta_fa"] * log_fa[i] + alpha
|
||||||
|
|
||||||
|
valid = np.isfinite(log_hedonic)
|
||||||
|
|
||||||
|
# Hold years and blend weight
|
||||||
|
sale_years = eligible["_sale_year"].to_numpy().astype(np.float64)
|
||||||
|
hold_years = np.maximum(CURRENT_YEAR - sale_years, 0.0)
|
||||||
|
blend_w = hold_years / (hold_years + tau)
|
||||||
|
|
||||||
|
# Blend in log space
|
||||||
|
log_index_est = np.log(
|
||||||
|
eligible["Estimated current price"].to_numpy().astype(np.float64)
|
||||||
|
)
|
||||||
|
log_blended = np.where(
|
||||||
|
valid,
|
||||||
|
(1 - blend_w) * log_index_est + blend_w * log_hedonic,
|
||||||
|
log_index_est,
|
||||||
|
)
|
||||||
|
blended_prices = np.exp(log_blended)
|
||||||
|
|
||||||
|
# Write back into df
|
||||||
|
eligible_indices = df.select(hedonic_mask).to_series().arg_true()
|
||||||
|
price_arr = df["Estimated current price"].to_numpy().astype(np.float64)
|
||||||
|
for i, idx in enumerate(eligible_indices):
|
||||||
|
price_arr[idx] = blended_prices[i]
|
||||||
|
df = df.with_columns(
|
||||||
|
pl.Series("Estimated current price", price_arr, dtype=pl.Float64),
|
||||||
|
)
|
||||||
|
|
||||||
|
n_blended = int(valid.sum())
|
||||||
|
avg_w = float(np.mean(blend_w[valid]))
|
||||||
|
print(
|
||||||
|
f" {n_blended:,} properties with hedonic blending (avg blend weight: {avg_w:.3f})"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
print(" No eligible properties for hedonic blending")
|
||||||
|
|
||||||
|
# Apply renovation premiums if provided
|
||||||
|
if args.renovation_premium is not None:
|
||||||
|
print("\nApplying renovation premiums...")
|
||||||
|
reno_prem = pl.read_parquet(args.renovation_premium)
|
||||||
|
print(f" Loaded {len(reno_prem):,} premium rows")
|
||||||
|
|
||||||
|
# Find properties with post-sale renovation events
|
||||||
|
has_reno = (
|
||||||
|
pl.col("renovation_history").is_not_null()
|
||||||
|
& (pl.col("renovation_history").list.len() > 0)
|
||||||
|
& pl.col("Estimated current price").is_not_null()
|
||||||
|
)
|
||||||
|
|
||||||
|
# Explode renovation events, filter to post-sale only
|
||||||
|
reno_rows = (
|
||||||
|
df.lazy()
|
||||||
|
.filter(has_reno)
|
||||||
|
.select("_sector", "_type_group", "_sale_year", "renovation_history")
|
||||||
|
.with_row_index("_row_idx")
|
||||||
|
.explode("renovation_history")
|
||||||
|
.with_columns(
|
||||||
|
pl.col("renovation_history").struct.field("year").alias("_event_year"),
|
||||||
|
pl.col("renovation_history").struct.field("event").alias("_event_type"),
|
||||||
|
)
|
||||||
|
.filter(pl.col("_event_year") > pl.col("_sale_year"))
|
||||||
|
.collect()
|
||||||
|
)
|
||||||
|
|
||||||
|
if len(reno_rows) > 0:
|
||||||
|
# Take most recent event per (row, event_type)
|
||||||
|
latest = (
|
||||||
|
reno_rows.lazy()
|
||||||
|
.group_by("_row_idx", "_event_type", "_sector", "_type_group")
|
||||||
|
.agg(pl.col("_event_year").max().alias("_event_year"))
|
||||||
|
.collect()
|
||||||
|
)
|
||||||
|
|
||||||
|
# Compute time-decayed premium
|
||||||
|
latest = latest.with_columns(
|
||||||
|
(-DECAY_RATE * (CURRENT_YEAR - pl.col("_event_year")).cast(pl.Float64))
|
||||||
|
.exp()
|
||||||
|
.alias("_decay"),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Join with renovation_premium.parquet — try typed first, fall back to "All"
|
||||||
|
rp_typed = reno_prem.filter(pl.col("type_group") != "All")
|
||||||
|
rp_all = reno_prem.filter(pl.col("type_group") == "All")
|
||||||
|
|
||||||
|
latest = (
|
||||||
|
latest.join(
|
||||||
|
rp_typed.select(
|
||||||
|
"sector",
|
||||||
|
"type_group",
|
||||||
|
"event_type",
|
||||||
|
pl.col("log_premium").alias("_lp_typed"),
|
||||||
|
),
|
||||||
|
left_on=["_sector", "_type_group", "_event_type"],
|
||||||
|
right_on=["sector", "type_group", "event_type"],
|
||||||
|
how="left",
|
||||||
|
)
|
||||||
|
.join(
|
||||||
|
rp_all.select(
|
||||||
|
"sector", "event_type", pl.col("log_premium").alias("_lp_all")
|
||||||
|
),
|
||||||
|
left_on=["_sector", "_event_type"],
|
||||||
|
right_on=["sector", "event_type"],
|
||||||
|
how="left",
|
||||||
|
)
|
||||||
|
.with_columns(
|
||||||
|
pl.col("_lp_typed")
|
||||||
|
.fill_null(pl.col("_lp_all"))
|
||||||
|
.fill_null(0.0)
|
||||||
|
.alias("_log_premium"),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
# Compute total decayed log premium per property
|
||||||
|
per_property = (
|
||||||
|
latest.lazy()
|
||||||
|
.with_columns(
|
||||||
|
(pl.col("_log_premium") * pl.col("_decay")).alias("_decayed_lp"),
|
||||||
|
)
|
||||||
|
.group_by("_row_idx")
|
||||||
|
.agg(pl.col("_decayed_lp").sum().alias("_reno_log_premium"))
|
||||||
|
.collect()
|
||||||
|
)
|
||||||
|
|
||||||
|
# We need to map _row_idx back to the main df. Re-derive the row indices.
|
||||||
|
# _row_idx was generated from filtered rows — we need the actual df row indices.
|
||||||
|
reno_mask = df.select(has_reno).to_series()
|
||||||
|
actual_indices = reno_mask.arg_true()
|
||||||
|
|
||||||
|
# Build a mapping: _row_idx -> actual df row
|
||||||
|
idx_map = per_property.with_columns(
|
||||||
|
pl.col("_row_idx")
|
||||||
|
.map_elements(
|
||||||
|
lambda i: int(actual_indices[i]),
|
||||||
|
return_dtype=pl.UInt32,
|
||||||
|
)
|
||||||
|
.alias("_df_row"),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create a full-length column of zeros, then fill in premium values
|
||||||
|
reno_log_prem = [0.0] * len(df)
|
||||||
|
for row in idx_map.iter_rows(named=True):
|
||||||
|
reno_log_prem[row["_df_row"]] = row["_reno_log_premium"]
|
||||||
|
|
||||||
|
df = df.with_columns(
|
||||||
|
pl.Series("_reno_log_premium", reno_log_prem, dtype=pl.Float64),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Apply: multiply estimated price by exp(reno_log_premium) where premium > 0
|
||||||
|
df = df.with_columns(
|
||||||
|
pl.when(pl.col("_reno_log_premium") != 0.0)
|
||||||
|
.then(
|
||||||
|
pl.col("Estimated current price")
|
||||||
|
* pl.col("_reno_log_premium").exp()
|
||||||
|
)
|
||||||
|
.otherwise(pl.col("Estimated current price"))
|
||||||
|
.alias("Estimated current price"),
|
||||||
|
)
|
||||||
|
|
||||||
|
n_with_premium = idx_map.height
|
||||||
|
avg_multiplier = math.exp(
|
||||||
|
per_property["_reno_log_premium"]
|
||||||
|
.filter(per_property["_reno_log_premium"] != 0.0)
|
||||||
|
.mean()
|
||||||
|
)
|
||||||
|
print(f" {n_with_premium:,} properties with renovation premium applied")
|
||||||
|
print(
|
||||||
|
f" Average premium multiplier: {avg_multiplier:.3f} ({avg_multiplier - 1:.1%} uplift)"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
print(" No properties with post-sale renovation events")
|
||||||
|
|
||||||
# Derive estimated price per sqm where both estimated price and floor area exist
|
# Derive estimated price per sqm where both estimated price and floor area exist
|
||||||
df = df.with_columns(
|
df = df.with_columns(
|
||||||
(pl.col("Estimated current price") / pl.col("Total floor area (sqm)"))
|
(pl.col("Estimated current price") / pl.col("Total floor area (sqm)"))
|
||||||
|
|
@ -135,20 +395,19 @@ def main():
|
||||||
.alias("Est. price per sqm"),
|
.alias("Est. price per sqm"),
|
||||||
)
|
)
|
||||||
|
|
||||||
n_adjusted = df.filter(
|
|
||||||
has_price & pl.col("_log_index_sale").is_not_null()
|
|
||||||
).height
|
|
||||||
n_with_price = df.filter(has_price).height
|
|
||||||
print(f" {n_adjusted:,} of {n_with_price:,} properties adjusted by index ({n_adjusted / max(n_with_price, 1) * 100:.1f}%)")
|
|
||||||
|
|
||||||
# Drop all temporary columns
|
# Drop all temporary columns
|
||||||
temp_cols = [c for c in df.columns if c.startswith("_") or c.startswith("log_idx_")]
|
temp_cols = [c for c in df.columns if c.startswith("_") or c.startswith("log_idx_")]
|
||||||
|
# Also drop hedonic-derived column if it was added
|
||||||
|
if "type_group" in df.columns:
|
||||||
|
temp_cols.append("type_group")
|
||||||
df = df.drop(temp_cols)
|
df = df.drop(temp_cols)
|
||||||
|
|
||||||
df.write_parquet(args.input)
|
df.write_parquet(args.input)
|
||||||
size_mb = args.input.stat().st_size / (1024 * 1024)
|
size_mb = args.input.stat().st_size / (1024 * 1024)
|
||||||
print(f"\nWrote {args.input} ({size_mb:.1f} MB)")
|
print(f"\nWrote {args.input} ({size_mb:.1f} MB)")
|
||||||
print(f" {len(df):,} rows, {len(df.columns)} columns (including 'Estimated current price')")
|
print(
|
||||||
|
f" {len(df):,} rows, {len(df.columns)} columns (including 'Estimated current price')"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|
|
||||||
|
|
@ -19,66 +19,38 @@ from scipy.sparse.linalg import lsqr
|
||||||
from scipy.spatial import KDTree
|
from scipy.spatial import KDTree
|
||||||
from tqdm import tqdm
|
from tqdm import tqdm
|
||||||
|
|
||||||
|
from pipeline.transform._price_utils import (
|
||||||
|
CURRENT_YEAR,
|
||||||
|
SHRINKAGE_K,
|
||||||
|
TYPE_GROUPS,
|
||||||
|
build_hedonic_features,
|
||||||
|
extract_centroids,
|
||||||
|
hierarchy_keys,
|
||||||
|
sector_expr,
|
||||||
|
type_group_expr,
|
||||||
|
)
|
||||||
|
|
||||||
# --- Constants ---
|
# --- Constants ---
|
||||||
MIN_PAIRS = 5
|
MIN_PAIRS = 5
|
||||||
SHRINKAGE_K = 50
|
|
||||||
OUTLIER_THRESHOLD = 3.0 # hard pre-filter; Huber handles the rest
|
OUTLIER_THRESHOLD = 3.0 # hard pre-filter; Huber handles the rest
|
||||||
HUBER_K = 1.345
|
HUBER_K = 1.345
|
||||||
IRLS_ITERATIONS = 5
|
IRLS_ITERATIONS = 5
|
||||||
SPATIAL_NEIGHBORS = 5
|
SPATIAL_NEIGHBORS = 5
|
||||||
SPATIAL_BLEND_K = 30
|
SPATIAL_BLEND_K = 30
|
||||||
CURRENT_YEAR = 2025
|
|
||||||
|
|
||||||
TYPE_GROUPS = ["Detached", "Semi-Detached", "Terraced", "Flats"]
|
|
||||||
TERRACE_TYPES = ["Mid-Terrace", "End-Terrace", "Enclosed Mid-Terrace", "Enclosed End-Terrace"]
|
|
||||||
AGE_BREAKS = [1900, 1930, 1950, 1967, 1983, 2000, 2010]
|
|
||||||
AGE_LABELS = ["pre-1900", "1900-1929", "1930-1949", "1950-1966", "1967-1982", "1983-1999", "2000-2009", "2010+"]
|
|
||||||
|
|
||||||
|
|
||||||
def type_group_expr():
|
|
||||||
"""Polars expression: Property type → type_group."""
|
|
||||||
return (
|
|
||||||
pl.when(pl.col("Property type").is_in(TERRACE_TYPES)).then(pl.lit("Terraced"))
|
|
||||||
.when(pl.col("Property type") == "Flats/Maisonettes").then(pl.lit("Flats"))
|
|
||||||
.when(pl.col("Property type").is_in(["Detached", "Semi-Detached"])).then(pl.col("Property type"))
|
|
||||||
.otherwise(pl.lit(None))
|
|
||||||
.alias("type_group")
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def age_band_expr():
|
|
||||||
"""Polars expression: Construction age (UInt16 year) → age band string."""
|
|
||||||
expr = pl.when(pl.col("Construction age").is_null()).then(pl.lit(None))
|
|
||||||
for i, brk in enumerate(AGE_BREAKS):
|
|
||||||
expr = expr.when(pl.col("Construction age") < brk).then(pl.lit(AGE_LABELS[i]))
|
|
||||||
return expr.otherwise(pl.lit(AGE_LABELS[-1])).alias("age_band")
|
|
||||||
|
|
||||||
|
|
||||||
def sector_expr():
|
|
||||||
"""Polars expression: Postcode → sector (drop last 2 chars, strip)."""
|
|
||||||
return pl.col("Postcode").str.slice(0, pl.col("Postcode").str.len_chars() - 2).str.strip_chars().alias("sector")
|
|
||||||
|
|
||||||
|
|
||||||
def hierarchy_keys(sector: str) -> tuple[str, str]:
|
|
||||||
"""Return (district, area) for a sector string."""
|
|
||||||
district = sector.rsplit(" ", 1)[0] if " " in sector else sector
|
|
||||||
area = ""
|
|
||||||
for ch in district:
|
|
||||||
if ch.isalpha():
|
|
||||||
area += ch
|
|
||||||
else:
|
|
||||||
break
|
|
||||||
return district, area
|
|
||||||
|
|
||||||
|
|
||||||
# --- Pair extraction ---
|
# --- Pair extraction ---
|
||||||
|
|
||||||
|
|
||||||
def extract_pairs(input_path: Path) -> pl.DataFrame:
|
def extract_pairs(input_path: Path) -> pl.DataFrame:
|
||||||
print("Extracting repeat-sale pairs...")
|
print("Extracting repeat-sale pairs...")
|
||||||
df = (
|
df = (
|
||||||
pl.scan_parquet(input_path)
|
pl.scan_parquet(input_path)
|
||||||
.select("Postcode", "historical_prices", "Property type")
|
.select("Postcode", "historical_prices", "Property type")
|
||||||
.filter(pl.col("Postcode").is_not_null(), pl.col("historical_prices").list.len() >= 2)
|
.filter(
|
||||||
|
pl.col("Postcode").is_not_null(),
|
||||||
|
pl.col("historical_prices").list.len() >= 2,
|
||||||
|
)
|
||||||
.with_columns(sector_expr(), type_group_expr())
|
.with_columns(sector_expr(), type_group_expr())
|
||||||
.collect()
|
.collect()
|
||||||
)
|
)
|
||||||
|
|
@ -87,7 +59,9 @@ def extract_pairs(input_path: Path) -> pl.DataFrame:
|
||||||
pairs = (
|
pairs = (
|
||||||
df.lazy()
|
df.lazy()
|
||||||
.with_columns(
|
.with_columns(
|
||||||
pl.col("historical_prices").list.slice(0, pl.col("historical_prices").list.len() - 1).alias("from_txn"),
|
pl.col("historical_prices")
|
||||||
|
.list.slice(0, pl.col("historical_prices").list.len() - 1)
|
||||||
|
.alias("from_txn"),
|
||||||
pl.col("historical_prices").list.slice(1).alias("to_txn"),
|
pl.col("historical_prices").list.slice(1).alias("to_txn"),
|
||||||
)
|
)
|
||||||
.explode("from_txn", "to_txn")
|
.explode("from_txn", "to_txn")
|
||||||
|
|
@ -98,10 +72,18 @@ def extract_pairs(input_path: Path) -> pl.DataFrame:
|
||||||
pl.col("to_txn").struct.field("price").alias("price2"),
|
pl.col("to_txn").struct.field("price").alias("price2"),
|
||||||
)
|
)
|
||||||
.select("sector", "type_group", "year1", "price1", "year2", "price2")
|
.select("sector", "type_group", "year1", "price1", "year2", "price2")
|
||||||
.filter(pl.col("price1") > 0, pl.col("price2") > 0, pl.col("year2") > pl.col("year1"))
|
.filter(
|
||||||
|
pl.col("price1") > 0,
|
||||||
|
pl.col("price2") > 0,
|
||||||
|
pl.col("year2") > pl.col("year1"),
|
||||||
|
)
|
||||||
.with_columns(
|
.with_columns(
|
||||||
(pl.col("price2").cast(pl.Float64) / pl.col("price1").cast(pl.Float64)).log().alias("log_ratio"),
|
(pl.col("price2").cast(pl.Float64) / pl.col("price1").cast(pl.Float64))
|
||||||
(1.0 / (pl.col("year2") - pl.col("year1")).cast(pl.Float64).sqrt()).alias("weight"),
|
.log()
|
||||||
|
.alias("log_ratio"),
|
||||||
|
(1.0 / (pl.col("year2") - pl.col("year1")).cast(pl.Float64).sqrt()).alias(
|
||||||
|
"weight"
|
||||||
|
),
|
||||||
)
|
)
|
||||||
.filter(pl.col("log_ratio").abs() <= OUTLIER_THRESHOLD)
|
.filter(pl.col("log_ratio").abs() <= OUTLIER_THRESHOLD)
|
||||||
.collect()
|
.collect()
|
||||||
|
|
@ -118,31 +100,14 @@ def extract_pairs(input_path: Path) -> pl.DataFrame:
|
||||||
return pairs
|
return pairs
|
||||||
|
|
||||||
|
|
||||||
# --- Sector centroids ---
|
|
||||||
|
|
||||||
def extract_centroids(input_path: Path) -> dict[str, tuple[float, float]]:
|
|
||||||
print("Computing sector centroids...")
|
|
||||||
df = (
|
|
||||||
pl.scan_parquet(input_path)
|
|
||||||
.select("Postcode", "lat", "lon")
|
|
||||||
.filter(pl.col("Postcode").is_not_null(), pl.col("lat").is_not_null())
|
|
||||||
.with_columns(sector_expr())
|
|
||||||
.group_by("sector")
|
|
||||||
.agg(pl.col("lat").mean(), pl.col("lon").mean())
|
|
||||||
.collect()
|
|
||||||
)
|
|
||||||
centroids = {}
|
|
||||||
for row in df.iter_rows(named=True):
|
|
||||||
centroids[row["sector"]] = (row["lat"], row["lon"])
|
|
||||||
print(f" {len(centroids):,} sector centroids")
|
|
||||||
return centroids
|
|
||||||
|
|
||||||
|
|
||||||
# --- Robust IRLS solver ---
|
# --- Robust IRLS solver ---
|
||||||
|
|
||||||
|
|
||||||
def solve_robust_index(
|
def solve_robust_index(
|
||||||
years1: np.ndarray, years2: np.ndarray,
|
years1: np.ndarray,
|
||||||
log_ratios: np.ndarray, base_weights: np.ndarray,
|
years2: np.ndarray,
|
||||||
|
log_ratios: np.ndarray,
|
||||||
|
base_weights: np.ndarray,
|
||||||
) -> dict[int, float]:
|
) -> dict[int, float]:
|
||||||
"""IRLS Huber M-estimation for the Case-Shiller repeat-sales model."""
|
"""IRLS Huber M-estimation for the Case-Shiller repeat-sales model."""
|
||||||
n = len(years1)
|
n = len(years1)
|
||||||
|
|
@ -205,11 +170,16 @@ def solve_robust_index(
|
||||||
def compute_indices_for_level(pairs: pl.DataFrame, group_col: str):
|
def compute_indices_for_level(pairs: pl.DataFrame, group_col: str):
|
||||||
"""Solve robust indices for each group. Returns (indices, n_pairs) dicts."""
|
"""Solve robust indices for each group. Returns (indices, n_pairs) dicts."""
|
||||||
groups = pairs.group_by(group_col).agg(
|
groups = pairs.group_by(group_col).agg(
|
||||||
pl.col("year1"), pl.col("year2"), pl.col("log_ratio"), pl.col("weight"),
|
pl.col("year1"),
|
||||||
|
pl.col("year2"),
|
||||||
|
pl.col("log_ratio"),
|
||||||
|
pl.col("weight"),
|
||||||
)
|
)
|
||||||
indices = {}
|
indices = {}
|
||||||
n_pairs = {}
|
n_pairs = {}
|
||||||
for row in tqdm(groups.iter_rows(named=True), total=len(groups), desc=f" {group_col}"):
|
for row in tqdm(
|
||||||
|
groups.iter_rows(named=True), total=len(groups), desc=f" {group_col}"
|
||||||
|
):
|
||||||
key = row[group_col]
|
key = row[group_col]
|
||||||
y1 = np.array(row["year1"], dtype=np.int32)
|
y1 = np.array(row["year1"], dtype=np.int32)
|
||||||
y2 = np.array(row["year2"], dtype=np.int32)
|
y2 = np.array(row["year2"], dtype=np.int32)
|
||||||
|
|
@ -224,28 +194,28 @@ def compute_indices_for_level(pairs: pl.DataFrame, group_col: str):
|
||||||
|
|
||||||
# --- Hedonic model ---
|
# --- Hedonic model ---
|
||||||
|
|
||||||
def compute_hedonic_index(input_path: Path, min_year: int, max_year: int) -> dict[int, float]:
|
|
||||||
|
def compute_hedonic_index(
|
||||||
|
input_path: Path, min_year: int, max_year: int
|
||||||
|
) -> dict[int, float]:
|
||||||
"""Two-step hedonic index: regress log(price) on features, average residual by year."""
|
"""Two-step hedonic index: regress log(price) on features, average residual by year."""
|
||||||
print("Computing hedonic index...")
|
print("Computing hedonic index...")
|
||||||
df = (
|
df = (
|
||||||
pl.scan_parquet(input_path)
|
pl.scan_parquet(input_path)
|
||||||
.select(
|
.select(
|
||||||
"Last known price", "Date of last transaction", "Property type",
|
"Last known price",
|
||||||
"Total floor area (sqm)", "Current energy rating",
|
"Date of last transaction",
|
||||||
"Number of bedrooms & living rooms", "Construction age",
|
"Property type",
|
||||||
|
"Total floor area (sqm)",
|
||||||
)
|
)
|
||||||
.filter(
|
.filter(
|
||||||
pl.col("Last known price").is_not_null(),
|
pl.col("Last known price").is_not_null(),
|
||||||
pl.col("Total floor area (sqm)").is_not_null(),
|
pl.col("Total floor area (sqm)").is_not_null(),
|
||||||
pl.col("Total floor area (sqm)") > 0,
|
pl.col("Total floor area (sqm)") > 0,
|
||||||
pl.col("Current energy rating").is_in(["A", "B", "C", "D", "E", "F", "G"]),
|
|
||||||
pl.col("Number of bedrooms & living rooms").is_not_null(),
|
|
||||||
pl.col("Construction age").is_not_null(),
|
|
||||||
)
|
)
|
||||||
.with_columns(
|
.with_columns(
|
||||||
pl.col("Date of last transaction").dt.year().alias("sale_year"),
|
pl.col("Date of last transaction").dt.year().alias("sale_year"),
|
||||||
type_group_expr(),
|
type_group_expr(),
|
||||||
age_band_expr(),
|
|
||||||
)
|
)
|
||||||
.filter(
|
.filter(
|
||||||
pl.col("type_group").is_not_null(),
|
pl.col("type_group").is_not_null(),
|
||||||
|
|
@ -261,29 +231,9 @@ def compute_hedonic_index(input_path: Path, min_year: int, max_year: int) -> dic
|
||||||
log_price = np.log(df["Last known price"].to_numpy().astype(np.float64))
|
log_price = np.log(df["Last known price"].to_numpy().astype(np.float64))
|
||||||
sale_years = df["sale_year"].to_numpy()
|
sale_years = df["sale_year"].to_numpy()
|
||||||
|
|
||||||
# Build feature matrix
|
# Build feature matrix (18 hedonic features + intercept)
|
||||||
parts = []
|
X = build_hedonic_features(df)
|
||||||
# log(floor_area)
|
F = np.hstack([X, np.ones((len(df), 1), dtype=np.float32)])
|
||||||
fa = df["Total floor area (sqm)"].to_numpy().astype(np.float32)
|
|
||||||
parts.append(np.log(np.maximum(fa, 1.0)).reshape(-1, 1))
|
|
||||||
# Type dummies (ref: Detached)
|
|
||||||
tg = df["type_group"].to_numpy()
|
|
||||||
for t in ["Terraced", "Semi-Detached", "Flats"]:
|
|
||||||
parts.append((tg == t).astype(np.float32).reshape(-1, 1))
|
|
||||||
# EPC dummies (ref: D)
|
|
||||||
epc = df["Current energy rating"].to_numpy()
|
|
||||||
for r in ["A", "B", "C", "E", "F", "G"]:
|
|
||||||
parts.append((epc == r).astype(np.float32).reshape(-1, 1))
|
|
||||||
# Rooms
|
|
||||||
parts.append(df["Number of bedrooms & living rooms"].to_numpy().astype(np.float32).reshape(-1, 1))
|
|
||||||
# Age band dummies (ref: pre-1900)
|
|
||||||
ab = df["age_band"].to_numpy()
|
|
||||||
for band in AGE_LABELS[1:]:
|
|
||||||
parts.append((ab == band).astype(np.float32).reshape(-1, 1))
|
|
||||||
# Intercept
|
|
||||||
parts.append(np.ones((len(df), 1), dtype=np.float32))
|
|
||||||
|
|
||||||
F = np.hstack(parts)
|
|
||||||
print(f" Feature matrix: {F.shape[0]:,} × {F.shape[1]}")
|
print(f" Feature matrix: {F.shape[0]:,} × {F.shape[1]}")
|
||||||
|
|
||||||
# Step 1: regress log(price) on features → quality score
|
# Step 1: regress log(price) on features → quality score
|
||||||
|
|
@ -303,12 +253,15 @@ def compute_hedonic_index(input_path: Path, min_year: int, max_year: int) -> dic
|
||||||
for y in hedonic:
|
for y in hedonic:
|
||||||
hedonic[y] -= base
|
hedonic[y] -= base
|
||||||
|
|
||||||
print(f" Hedonic index: {len(hedonic)} years, range {min(hedonic.values()):.3f} to {max(hedonic.values()):.3f}")
|
print(
|
||||||
|
f" Hedonic index: {len(hedonic)} years, range {min(hedonic.values()):.3f} to {max(hedonic.values()):.3f}"
|
||||||
|
)
|
||||||
return hedonic
|
return hedonic
|
||||||
|
|
||||||
|
|
||||||
# --- Shrinkage ---
|
# --- Shrinkage ---
|
||||||
|
|
||||||
|
|
||||||
def shrink_index(raw: dict, parent: dict, n_pairs: int, k: int = SHRINKAGE_K) -> dict:
|
def shrink_index(raw: dict, parent: dict, n_pairs: int, k: int = SHRINKAGE_K) -> dict:
|
||||||
w = n_pairs / (n_pairs + k)
|
w = n_pairs / (n_pairs + k)
|
||||||
result = {}
|
result = {}
|
||||||
|
|
@ -320,9 +273,18 @@ def shrink_index(raw: dict, parent: dict, n_pairs: int, k: int = SHRINKAGE_K) ->
|
||||||
|
|
||||||
|
|
||||||
def apply_shrinkage(
|
def apply_shrinkage(
|
||||||
sector_idx, sector_n, district_idx, district_n,
|
sector_idx,
|
||||||
area_idx, area_n, national_idx, national_n,
|
sector_n,
|
||||||
hedonic_idx, all_sectors, sector_to_dist, dist_to_area,
|
district_idx,
|
||||||
|
district_n,
|
||||||
|
area_idx,
|
||||||
|
area_n,
|
||||||
|
national_idx,
|
||||||
|
national_n,
|
||||||
|
hedonic_idx,
|
||||||
|
all_sectors,
|
||||||
|
sector_to_dist,
|
||||||
|
dist_to_area,
|
||||||
):
|
):
|
||||||
"""Top-down hierarchical shrinkage: national→hedonic, area→national, etc."""
|
"""Top-down hierarchical shrinkage: national→hedonic, area→national, etc."""
|
||||||
# National → hedonic
|
# National → hedonic
|
||||||
|
|
@ -361,8 +323,11 @@ def apply_shrinkage(
|
||||||
|
|
||||||
# --- Spatial smoothing ---
|
# --- Spatial smoothing ---
|
||||||
|
|
||||||
|
|
||||||
def spatial_smooth(
|
def spatial_smooth(
|
||||||
sector_indices: dict, centroids: dict, n_pairs_map: dict,
|
sector_indices: dict,
|
||||||
|
centroids: dict,
|
||||||
|
n_pairs_map: dict,
|
||||||
) -> dict:
|
) -> dict:
|
||||||
"""Blend sparse sector indices with K nearest neighbors."""
|
"""Blend sparse sector indices with K nearest neighbors."""
|
||||||
# Build coordinate arrays for sectors with centroids
|
# Build coordinate arrays for sectors with centroids
|
||||||
|
|
@ -420,6 +385,7 @@ def spatial_smooth(
|
||||||
|
|
||||||
# --- Forward fill ---
|
# --- Forward fill ---
|
||||||
|
|
||||||
|
|
||||||
def forward_fill(index: dict, min_year: int, max_year: int) -> dict:
|
def forward_fill(index: dict, min_year: int, max_year: int) -> dict:
|
||||||
filled = {}
|
filled = {}
|
||||||
last = 0.0
|
last = 0.0
|
||||||
|
|
@ -432,8 +398,11 @@ def forward_fill(index: dict, min_year: int, max_year: int) -> dict:
|
||||||
|
|
||||||
# --- Main ---
|
# --- Main ---
|
||||||
|
|
||||||
|
|
||||||
def main():
|
def main():
|
||||||
parser = argparse.ArgumentParser(description="Build improved repeat-sales price index")
|
parser = argparse.ArgumentParser(
|
||||||
|
description="Build improved repeat-sales price index"
|
||||||
|
)
|
||||||
parser.add_argument("--input", type=Path, required=True)
|
parser.add_argument("--input", type=Path, required=True)
|
||||||
parser.add_argument("--output", type=Path, required=True)
|
parser.add_argument("--output", type=Path, required=True)
|
||||||
args = parser.parse_args()
|
args = parser.parse_args()
|
||||||
|
|
@ -474,8 +443,10 @@ def main():
|
||||||
# National
|
# National
|
||||||
np_arrs = typed.select("year1", "year2", "log_ratio", "weight")
|
np_arrs = typed.select("year1", "year2", "log_ratio", "weight")
|
||||||
national_idx = solve_robust_index(
|
national_idx = solve_robust_index(
|
||||||
np_arrs["year1"].to_numpy(), np_arrs["year2"].to_numpy(),
|
np_arrs["year1"].to_numpy(),
|
||||||
np_arrs["log_ratio"].to_numpy(), np_arrs["weight"].to_numpy(),
|
np_arrs["year2"].to_numpy(),
|
||||||
|
np_arrs["log_ratio"].to_numpy(),
|
||||||
|
np_arrs["weight"].to_numpy(),
|
||||||
)
|
)
|
||||||
national_n = len(typed)
|
national_n = len(typed)
|
||||||
print(f" National: {len(national_idx)} years")
|
print(f" National: {len(national_idx)} years")
|
||||||
|
|
@ -485,14 +456,25 @@ def main():
|
||||||
area_idx, area_n = compute_indices_for_level(typed, "area")
|
area_idx, area_n = compute_indices_for_level(typed, "area")
|
||||||
district_idx, district_n = compute_indices_for_level(typed, "district")
|
district_idx, district_n = compute_indices_for_level(typed, "district")
|
||||||
sector_idx, sector_n = compute_indices_for_level(typed, "sector")
|
sector_idx, sector_n = compute_indices_for_level(typed, "sector")
|
||||||
print(f" {len(area_idx)} areas, {len(district_idx)} districts, {len(sector_idx)} sectors")
|
print(
|
||||||
|
f" {len(area_idx)} areas, {len(district_idx)} districts, {len(sector_idx)} sectors"
|
||||||
|
)
|
||||||
|
|
||||||
# Shrinkage
|
# Shrinkage
|
||||||
print(" Applying shrinkage...")
|
print(" Applying shrinkage...")
|
||||||
sector_shrunk = apply_shrinkage(
|
sector_shrunk = apply_shrinkage(
|
||||||
sector_idx, sector_n, district_idx, district_n,
|
sector_idx,
|
||||||
area_idx, area_n, national_idx, national_n,
|
sector_n,
|
||||||
hedonic_idx, all_sectors, sector_to_dist, dist_to_area,
|
district_idx,
|
||||||
|
district_n,
|
||||||
|
area_idx,
|
||||||
|
area_n,
|
||||||
|
national_idx,
|
||||||
|
national_n,
|
||||||
|
hedonic_idx,
|
||||||
|
all_sectors,
|
||||||
|
sector_to_dist,
|
||||||
|
dist_to_area,
|
||||||
)
|
)
|
||||||
|
|
||||||
# Spatial smoothing
|
# Spatial smoothing
|
||||||
|
|
@ -519,15 +501,22 @@ def main():
|
||||||
|
|
||||||
result = pl.DataFrame(
|
result = pl.DataFrame(
|
||||||
rows,
|
rows,
|
||||||
schema={"sector": pl.String, "type_group": pl.String, "year": pl.Int32,
|
schema={
|
||||||
"log_index": pl.Float64, "n_pairs": pl.Int64},
|
"sector": pl.String,
|
||||||
|
"type_group": pl.String,
|
||||||
|
"year": pl.Int32,
|
||||||
|
"log_index": pl.Float64,
|
||||||
|
"n_pairs": pl.Int64,
|
||||||
|
},
|
||||||
orient="row",
|
orient="row",
|
||||||
).sort("type_group", "sector", "year")
|
).sort("type_group", "sector", "year")
|
||||||
|
|
||||||
result.write_parquet(args.output)
|
result.write_parquet(args.output)
|
||||||
size_mb = args.output.stat().st_size / (1024 * 1024)
|
size_mb = args.output.stat().st_size / (1024 * 1024)
|
||||||
print(f"\nWrote {args.output} ({size_mb:.1f} MB)")
|
print(f"\nWrote {args.output} ({size_mb:.1f} MB)")
|
||||||
print(f" {result['sector'].n_unique():,} sectors × {len(all_type_groups)} types × {max_year - min_year + 1} years = {len(result):,} rows")
|
print(
|
||||||
|
f" {result['sector'].n_unique():,} sectors × {len(all_type_groups)} types × {max_year - min_year + 1} years = {len(result):,} rows"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
|
|
|
||||||
572
pipeline/transform/renovation_premium.py
Normal file
572
pipeline/transform/renovation_premium.py
Normal file
|
|
@ -0,0 +1,572 @@
|
||||||
|
"""Estimate per-area renovation premiums from repeat-sale residuals.
|
||||||
|
|
||||||
|
For each repeat-sale pair, computes the residual after removing the price-index
|
||||||
|
predicted return. Pairs where renovation events occurred between sales should have
|
||||||
|
systematically higher residuals. A WLS regression estimates the log-premium per
|
||||||
|
event type, with hierarchical shrinkage and spatial smoothing.
|
||||||
|
|
||||||
|
Output: renovation_premium.parquet — sector × type_group × event_type → log_premium
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import math
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import numpy as np
|
||||||
|
import polars as pl
|
||||||
|
from scipy.spatial import KDTree
|
||||||
|
|
||||||
|
from pipeline.transform._price_utils import (
|
||||||
|
SHRINKAGE_K,
|
||||||
|
TYPE_GROUPS,
|
||||||
|
extract_centroids,
|
||||||
|
hierarchy_keys,
|
||||||
|
sector_expr,
|
||||||
|
type_group_expr,
|
||||||
|
)
|
||||||
|
|
||||||
|
HALF_LIFE = 10.0
|
||||||
|
DECAY_RATE = math.log(2) / HALF_LIFE
|
||||||
|
OUTLIER_THRESHOLD = 3.0
|
||||||
|
MIN_PAIRS = 10
|
||||||
|
SPATIAL_NEIGHBORS = 5
|
||||||
|
SPATIAL_BLEND_K = 30
|
||||||
|
EVENT_TYPES = ["Extension", "Renovation", "Remodeling"]
|
||||||
|
|
||||||
|
|
||||||
|
def extract_pairs_with_events(input_path: Path, index_path: Path) -> pl.DataFrame:
|
||||||
|
"""Extract repeat-sale pairs with renovation events and index residuals."""
|
||||||
|
print("Extracting repeat-sale pairs with renovation events...")
|
||||||
|
|
||||||
|
df = (
|
||||||
|
pl.scan_parquet(input_path)
|
||||||
|
.select("Postcode", "historical_prices", "Property type", "renovation_history")
|
||||||
|
.filter(
|
||||||
|
pl.col("Postcode").is_not_null(),
|
||||||
|
pl.col("historical_prices").list.len() >= 2,
|
||||||
|
)
|
||||||
|
.with_columns(sector_expr(), type_group_expr())
|
||||||
|
.collect()
|
||||||
|
)
|
||||||
|
print(f" {len(df):,} properties with 2+ transactions")
|
||||||
|
|
||||||
|
# Build consecutive pairs
|
||||||
|
pairs = (
|
||||||
|
df.lazy()
|
||||||
|
.with_columns(
|
||||||
|
pl.col("historical_prices")
|
||||||
|
.list.slice(0, pl.col("historical_prices").list.len() - 1)
|
||||||
|
.alias("from_txn"),
|
||||||
|
pl.col("historical_prices").list.slice(1).alias("to_txn"),
|
||||||
|
)
|
||||||
|
.explode("from_txn", "to_txn")
|
||||||
|
.with_columns(
|
||||||
|
pl.col("from_txn").struct.field("year").alias("year1"),
|
||||||
|
pl.col("from_txn").struct.field("price").alias("price1"),
|
||||||
|
pl.col("to_txn").struct.field("year").alias("year2"),
|
||||||
|
pl.col("to_txn").struct.field("price").alias("price2"),
|
||||||
|
)
|
||||||
|
.select(
|
||||||
|
"sector",
|
||||||
|
"type_group",
|
||||||
|
"year1",
|
||||||
|
"price1",
|
||||||
|
"year2",
|
||||||
|
"price2",
|
||||||
|
"renovation_history",
|
||||||
|
)
|
||||||
|
.filter(
|
||||||
|
pl.col("price1") > 0,
|
||||||
|
pl.col("price2") > 0,
|
||||||
|
pl.col("year2") > pl.col("year1"),
|
||||||
|
)
|
||||||
|
.with_columns(
|
||||||
|
(pl.col("price2").cast(pl.Float64) / pl.col("price1").cast(pl.Float64))
|
||||||
|
.log()
|
||||||
|
.alias("log_ratio"),
|
||||||
|
)
|
||||||
|
.filter(pl.col("log_ratio").abs() <= OUTLIER_THRESHOLD)
|
||||||
|
.collect()
|
||||||
|
)
|
||||||
|
print(f" {len(pairs):,} repeat-sale pairs")
|
||||||
|
|
||||||
|
# Join price index to compute residuals
|
||||||
|
index = pl.read_parquet(index_path)
|
||||||
|
has_type_group = "type_group" in index.columns
|
||||||
|
|
||||||
|
if has_type_group:
|
||||||
|
idx_typed = index.filter(pl.col("type_group") != "All")
|
||||||
|
idx_all = index.filter(pl.col("type_group") == "All")
|
||||||
|
|
||||||
|
# Join at year1
|
||||||
|
pairs = pairs.join(
|
||||||
|
idx_typed.select(
|
||||||
|
"sector", "type_group", "year", pl.col("log_index").alias("li1_typed")
|
||||||
|
),
|
||||||
|
left_on=["sector", "type_group", "year1"],
|
||||||
|
right_on=["sector", "type_group", "year"],
|
||||||
|
how="left",
|
||||||
|
).join(
|
||||||
|
idx_all.select("sector", "year", pl.col("log_index").alias("li1_all")),
|
||||||
|
left_on=["sector", "year1"],
|
||||||
|
right_on=["sector", "year"],
|
||||||
|
how="left",
|
||||||
|
)
|
||||||
|
# Join at year2
|
||||||
|
pairs = pairs.join(
|
||||||
|
idx_typed.select(
|
||||||
|
"sector", "type_group", "year", pl.col("log_index").alias("li2_typed")
|
||||||
|
),
|
||||||
|
left_on=["sector", "type_group", "year2"],
|
||||||
|
right_on=["sector", "type_group", "year"],
|
||||||
|
how="left",
|
||||||
|
).join(
|
||||||
|
idx_all.select("sector", "year", pl.col("log_index").alias("li2_all")),
|
||||||
|
left_on=["sector", "year2"],
|
||||||
|
right_on=["sector", "year"],
|
||||||
|
how="left",
|
||||||
|
)
|
||||||
|
|
||||||
|
pairs = pairs.with_columns(
|
||||||
|
(pl.col("li1_typed").fill_null(pl.col("li1_all"))).alias("_li1"),
|
||||||
|
(pl.col("li2_typed").fill_null(pl.col("li2_all"))).alias("_li2"),
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
pairs = pairs.join(
|
||||||
|
index.select("sector", "year", pl.col("log_index").alias("_li1")),
|
||||||
|
left_on=["sector", "year1"],
|
||||||
|
right_on=["sector", "year"],
|
||||||
|
how="left",
|
||||||
|
).join(
|
||||||
|
index.select("sector", "year", pl.col("log_index").alias("_li2")),
|
||||||
|
left_on=["sector", "year2"],
|
||||||
|
right_on=["sector", "year"],
|
||||||
|
how="left",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Compute residual = log_ratio - (index2 - index1)
|
||||||
|
pairs = pairs.with_columns(
|
||||||
|
(
|
||||||
|
pl.col("log_ratio")
|
||||||
|
- (pl.col("_li2").fill_null(0.0) - pl.col("_li1").fill_null(0.0))
|
||||||
|
).alias("residual"),
|
||||||
|
(1.0 / (pl.col("year2") - pl.col("year1")).cast(pl.Float64).sqrt()).alias(
|
||||||
|
"weight"
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
# For each pair, compute time-decayed renovation indicators
|
||||||
|
# Use row index for unique identification (composite keys aren't unique per pair)
|
||||||
|
pairs = pairs.with_row_index("_pair_idx")
|
||||||
|
|
||||||
|
for et in EVENT_TYPES:
|
||||||
|
col_name = f"has_{et.lower()}"
|
||||||
|
pairs = pairs.with_columns(pl.lit(0.0).alias(col_name))
|
||||||
|
|
||||||
|
# Process properties that have renovation history
|
||||||
|
has_reno = pairs.filter(
|
||||||
|
pl.col("renovation_history").is_not_null()
|
||||||
|
& (pl.col("renovation_history").list.len() > 0)
|
||||||
|
)
|
||||||
|
|
||||||
|
if len(has_reno) > 0:
|
||||||
|
reno_exploded = (
|
||||||
|
has_reno.select("_pair_idx", "year1", "year2", "renovation_history")
|
||||||
|
.explode("renovation_history")
|
||||||
|
.with_columns(
|
||||||
|
pl.col("renovation_history").struct.field("year").alias("event_year"),
|
||||||
|
pl.col("renovation_history").struct.field("event").alias("event_type"),
|
||||||
|
)
|
||||||
|
# Only events between the two sales
|
||||||
|
.filter(
|
||||||
|
(pl.col("event_year") > pl.col("year1"))
|
||||||
|
& (pl.col("event_year") <= pl.col("year2"))
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
if len(reno_exploded) > 0:
|
||||||
|
# For each pair + event type, take the most recent event
|
||||||
|
latest_events = reno_exploded.group_by(
|
||||||
|
"_pair_idx", "event_type", "year2"
|
||||||
|
).agg(pl.col("event_year").max().alias("latest_event_year"))
|
||||||
|
|
||||||
|
# Compute time-decayed indicator: exp(-decay_rate * (year2 - event_year))
|
||||||
|
latest_events = latest_events.with_columns(
|
||||||
|
(
|
||||||
|
-DECAY_RATE
|
||||||
|
* (pl.col("year2") - pl.col("latest_event_year")).cast(pl.Float64)
|
||||||
|
)
|
||||||
|
.exp()
|
||||||
|
.alias("decayed_indicator"),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Pivot to wide format using _pair_idx for unique join
|
||||||
|
for et in EVENT_TYPES:
|
||||||
|
et_data = latest_events.filter(pl.col("event_type") == et)
|
||||||
|
if len(et_data) > 0:
|
||||||
|
col_name = f"has_{et.lower()}"
|
||||||
|
pairs = (
|
||||||
|
pairs.join(
|
||||||
|
et_data.select(
|
||||||
|
"_pair_idx",
|
||||||
|
pl.col("decayed_indicator").alias(f"_{col_name}"),
|
||||||
|
),
|
||||||
|
on="_pair_idx",
|
||||||
|
how="left",
|
||||||
|
)
|
||||||
|
.with_columns(
|
||||||
|
pl.col(f"_{col_name}").fill_null(0.0).alias(col_name),
|
||||||
|
)
|
||||||
|
.drop(f"_{col_name}")
|
||||||
|
)
|
||||||
|
|
||||||
|
pairs = pairs.drop("_pair_idx")
|
||||||
|
|
||||||
|
# Add hierarchy columns
|
||||||
|
pairs = pairs.with_columns(
|
||||||
|
pl.col("sector").str.replace(r"\s+\d+$", "").alias("district"),
|
||||||
|
).with_columns(
|
||||||
|
pl.col("district").str.replace(r"\d.*$", "").alias("area"),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Count reno pairs
|
||||||
|
reno_mask = (
|
||||||
|
(pl.col("has_extension") > 0)
|
||||||
|
| (pl.col("has_renovation") > 0)
|
||||||
|
| (pl.col("has_remodeling") > 0)
|
||||||
|
)
|
||||||
|
n_reno = pairs.filter(reno_mask).height
|
||||||
|
print(
|
||||||
|
f" {n_reno:,} pairs with renovation events ({n_reno / len(pairs) * 100:.1f}%)"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Drop temporary columns from index join + renovation_history (no longer needed)
|
||||||
|
temp_cols = [
|
||||||
|
c
|
||||||
|
for c in pairs.columns
|
||||||
|
if c.startswith("_li") or c.startswith("li1_") or c.startswith("li2_")
|
||||||
|
]
|
||||||
|
pairs = pairs.drop(temp_cols + ["renovation_history"])
|
||||||
|
|
||||||
|
return pairs
|
||||||
|
|
||||||
|
|
||||||
|
def wls_regression(
|
||||||
|
residuals: np.ndarray,
|
||||||
|
weights: np.ndarray,
|
||||||
|
X: np.ndarray,
|
||||||
|
) -> np.ndarray:
|
||||||
|
"""Weighted least squares: residual ~ X (with intercept column in X).
|
||||||
|
|
||||||
|
Uses sqrt(weights) scaling to avoid building a full N×N diagonal matrix.
|
||||||
|
"""
|
||||||
|
sqrt_w = np.sqrt(weights)[:, np.newaxis]
|
||||||
|
Xw = X * sqrt_w
|
||||||
|
yw = residuals * sqrt_w.ravel()
|
||||||
|
try:
|
||||||
|
betas = np.linalg.lstsq(Xw, yw, rcond=None)[0]
|
||||||
|
except np.linalg.LinAlgError:
|
||||||
|
betas = np.zeros(X.shape[1])
|
||||||
|
return betas
|
||||||
|
|
||||||
|
|
||||||
|
def compute_premiums_for_group(df: pl.DataFrame) -> dict[str, float]:
|
||||||
|
"""Run WLS regression for a group, return {event_type: log_premium}."""
|
||||||
|
n = len(df)
|
||||||
|
if n < MIN_PAIRS:
|
||||||
|
return {}
|
||||||
|
|
||||||
|
residuals = df["residual"].to_numpy().astype(np.float64)
|
||||||
|
weights = df["weight"].to_numpy().astype(np.float64)
|
||||||
|
|
||||||
|
# Build design matrix: intercept + 3 event indicators
|
||||||
|
X = np.column_stack(
|
||||||
|
[
|
||||||
|
np.ones(n),
|
||||||
|
df["has_extension"].to_numpy().astype(np.float64),
|
||||||
|
df["has_renovation"].to_numpy().astype(np.float64),
|
||||||
|
df["has_remodeling"].to_numpy().astype(np.float64),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
# Check if we have any renovation pairs in this group
|
||||||
|
reno_sum = X[:, 1:].sum()
|
||||||
|
if reno_sum < 1.0:
|
||||||
|
return {}
|
||||||
|
|
||||||
|
betas = wls_regression(residuals, weights, X)
|
||||||
|
# betas[0] is intercept, betas[1:4] are the premiums
|
||||||
|
return {
|
||||||
|
"Extension": float(betas[1]),
|
||||||
|
"Renovation": float(betas[2]),
|
||||||
|
"Remodeling": float(betas[3]),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def compute_premiums_for_level(
|
||||||
|
pairs: pl.DataFrame, group_col: str
|
||||||
|
) -> tuple[dict, dict]:
|
||||||
|
"""Compute premiums per group at a given hierarchy level.
|
||||||
|
|
||||||
|
Returns (premiums, n_reno_pairs) dicts keyed by group value.
|
||||||
|
premiums[key] = {event_type: log_premium}
|
||||||
|
"""
|
||||||
|
groups = pairs.group_by(group_col)
|
||||||
|
premiums = {}
|
||||||
|
n_reno_pairs = {}
|
||||||
|
for key, group_df in groups:
|
||||||
|
key_val = key[0]
|
||||||
|
result = compute_premiums_for_group(group_df)
|
||||||
|
if result:
|
||||||
|
premiums[key_val] = result
|
||||||
|
# Count pairs with any reno indicator
|
||||||
|
reno_mask = (
|
||||||
|
(group_df["has_extension"].to_numpy() > 0)
|
||||||
|
| (group_df["has_renovation"].to_numpy() > 0)
|
||||||
|
| (group_df["has_remodeling"].to_numpy() > 0)
|
||||||
|
)
|
||||||
|
n_reno_pairs[key_val] = int(reno_mask.sum())
|
||||||
|
return premiums, n_reno_pairs
|
||||||
|
|
||||||
|
|
||||||
|
def shrink_premium(
|
||||||
|
raw: dict[str, float], parent: dict[str, float], n: int
|
||||||
|
) -> dict[str, float]:
|
||||||
|
"""Shrink raw premiums toward parent level."""
|
||||||
|
w = n / (n + SHRINKAGE_K)
|
||||||
|
result = {}
|
||||||
|
for et in EVENT_TYPES:
|
||||||
|
r = raw.get(et, parent.get(et, 0.0))
|
||||||
|
p = parent.get(et, raw.get(et, 0.0))
|
||||||
|
result[et] = w * r + (1 - w) * p
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def apply_shrinkage(
|
||||||
|
sector_prem,
|
||||||
|
sector_n,
|
||||||
|
district_prem,
|
||||||
|
district_n,
|
||||||
|
area_prem,
|
||||||
|
area_n,
|
||||||
|
national_prem,
|
||||||
|
national_n,
|
||||||
|
all_sectors,
|
||||||
|
sector_to_dist,
|
||||||
|
dist_to_area,
|
||||||
|
):
|
||||||
|
"""Top-down hierarchical shrinkage for premiums."""
|
||||||
|
# Area -> national
|
||||||
|
area_shrunk = {}
|
||||||
|
for area, prem in area_prem.items():
|
||||||
|
area_shrunk[area] = shrink_premium(prem, national_prem, area_n.get(area, 0))
|
||||||
|
|
||||||
|
# District -> area
|
||||||
|
district_shrunk = {}
|
||||||
|
for dist, prem in district_prem.items():
|
||||||
|
a = dist_to_area.get(dist, "")
|
||||||
|
parent = area_shrunk.get(a, national_prem)
|
||||||
|
district_shrunk[dist] = shrink_premium(prem, parent, district_n.get(dist, 0))
|
||||||
|
|
||||||
|
# Sector -> district
|
||||||
|
sector_shrunk = {}
|
||||||
|
for sec, prem in sector_prem.items():
|
||||||
|
d = sector_to_dist.get(sec, "")
|
||||||
|
parent = district_shrunk.get(d, national_prem)
|
||||||
|
sector_shrunk[sec] = shrink_premium(prem, parent, sector_n.get(sec, 0))
|
||||||
|
|
||||||
|
# Fill missing sectors
|
||||||
|
for sec in all_sectors:
|
||||||
|
if sec not in sector_shrunk:
|
||||||
|
d = sector_to_dist.get(sec, "")
|
||||||
|
a = dist_to_area.get(d, "")
|
||||||
|
sector_shrunk[sec] = district_shrunk.get(
|
||||||
|
d, area_shrunk.get(a, national_prem)
|
||||||
|
)
|
||||||
|
|
||||||
|
return sector_shrunk
|
||||||
|
|
||||||
|
|
||||||
|
def spatial_smooth(
|
||||||
|
sector_premiums: dict[str, dict[str, float]],
|
||||||
|
centroids: dict[str, tuple[float, float]],
|
||||||
|
n_reno_map: dict[str, int],
|
||||||
|
) -> dict[str, dict[str, float]]:
|
||||||
|
"""Blend sparse sector premiums with K nearest neighbors."""
|
||||||
|
sectors_with_coords = [s for s in sector_premiums if s in centroids]
|
||||||
|
if len(sectors_with_coords) < SPATIAL_NEIGHBORS + 1:
|
||||||
|
return sector_premiums
|
||||||
|
|
||||||
|
coords = np.array([centroids[s] for s in sectors_with_coords])
|
||||||
|
mean_lat = np.mean(coords[:, 0])
|
||||||
|
scale = np.cos(np.radians(mean_lat))
|
||||||
|
scaled_coords = np.column_stack([coords[:, 0], coords[:, 1] * scale])
|
||||||
|
tree = KDTree(scaled_coords)
|
||||||
|
|
||||||
|
result = dict(sector_premiums)
|
||||||
|
for i, sec in enumerate(sectors_with_coords):
|
||||||
|
n = n_reno_map.get(sec, 0)
|
||||||
|
self_w = n / (n + SPATIAL_BLEND_K)
|
||||||
|
if self_w > 0.95:
|
||||||
|
continue
|
||||||
|
|
||||||
|
dists, idxs = tree.query(scaled_coords[i], k=SPATIAL_NEIGHBORS + 1)
|
||||||
|
neighbor_dists = dists[1:]
|
||||||
|
neighbor_idxs = idxs[1:]
|
||||||
|
|
||||||
|
inv_dists = []
|
||||||
|
neighbor_prems = []
|
||||||
|
for d, j in zip(neighbor_dists, neighbor_idxs):
|
||||||
|
ns = sectors_with_coords[j]
|
||||||
|
if d > 0 and ns in sector_premiums:
|
||||||
|
inv_dists.append(1.0 / d)
|
||||||
|
neighbor_prems.append(sector_premiums[ns])
|
||||||
|
|
||||||
|
if not neighbor_prems:
|
||||||
|
continue
|
||||||
|
|
||||||
|
total_inv = sum(inv_dists)
|
||||||
|
nbr_w = 1.0 - self_w
|
||||||
|
ws = [iw / total_inv * nbr_w for iw in inv_dists]
|
||||||
|
|
||||||
|
blended = {}
|
||||||
|
for et in EVENT_TYPES:
|
||||||
|
val = self_w * sector_premiums[sec].get(et, 0.0)
|
||||||
|
for np_dict, w in zip(neighbor_prems, ws):
|
||||||
|
val += w * np_dict.get(et, 0.0)
|
||||||
|
blended[et] = val
|
||||||
|
result[sec] = blended
|
||||||
|
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description="Estimate renovation premiums from repeat-sale residuals"
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--input", type=Path, required=True, help="Path to wide.parquet"
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--index", type=Path, required=True, help="Path to price_index.parquet"
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--output", type=Path, required=True, help="Output renovation_premium.parquet"
|
||||||
|
)
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
pairs = extract_pairs_with_events(args.input, args.index)
|
||||||
|
centroids = extract_centroids(args.input)
|
||||||
|
|
||||||
|
# Precompute hierarchy
|
||||||
|
all_sectors = pairs["sector"].unique().to_list()
|
||||||
|
sector_to_dist = {}
|
||||||
|
dist_to_area = {}
|
||||||
|
for s in all_sectors:
|
||||||
|
d, a = hierarchy_keys(s)
|
||||||
|
sector_to_dist[s] = d
|
||||||
|
dist_to_area[d] = a
|
||||||
|
|
||||||
|
all_type_groups = ["All"] + TYPE_GROUPS
|
||||||
|
rows = []
|
||||||
|
|
||||||
|
for tg in all_type_groups:
|
||||||
|
print(f"\n--- {tg} ---")
|
||||||
|
typed = pairs if tg == "All" else pairs.filter(pl.col("type_group") == tg)
|
||||||
|
if len(typed) < MIN_PAIRS:
|
||||||
|
print(f" Skipping (only {len(typed)} pairs)")
|
||||||
|
continue
|
||||||
|
|
||||||
|
print(f" {len(typed):,} pairs")
|
||||||
|
|
||||||
|
# National
|
||||||
|
national_prem = compute_premiums_for_group(typed)
|
||||||
|
national_reno = typed.filter(
|
||||||
|
(pl.col("has_extension") > 0)
|
||||||
|
| (pl.col("has_renovation") > 0)
|
||||||
|
| (pl.col("has_remodeling") > 0)
|
||||||
|
).height
|
||||||
|
if not national_prem:
|
||||||
|
print(" No renovation pairs at national level, skipping")
|
||||||
|
continue
|
||||||
|
|
||||||
|
print(
|
||||||
|
" National premiums: "
|
||||||
|
+ ", ".join(
|
||||||
|
f"{et}: {v:.4f} ({math.exp(v) - 1:.1%})"
|
||||||
|
for et, v in national_prem.items()
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
# Per-level
|
||||||
|
print(" Computing per-level premiums:")
|
||||||
|
area_prem, area_n = compute_premiums_for_level(typed, "area")
|
||||||
|
district_prem, district_n = compute_premiums_for_level(typed, "district")
|
||||||
|
sector_prem, sector_n = compute_premiums_for_level(typed, "sector")
|
||||||
|
print(
|
||||||
|
f" {len(area_prem)} areas, {len(district_prem)} districts, {len(sector_prem)} sectors with data"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Shrinkage
|
||||||
|
print(" Applying shrinkage...")
|
||||||
|
sector_shrunk = apply_shrinkage(
|
||||||
|
sector_prem,
|
||||||
|
sector_n,
|
||||||
|
district_prem,
|
||||||
|
district_n,
|
||||||
|
area_prem,
|
||||||
|
area_n,
|
||||||
|
national_prem,
|
||||||
|
national_reno,
|
||||||
|
all_sectors,
|
||||||
|
sector_to_dist,
|
||||||
|
dist_to_area,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Spatial smoothing
|
||||||
|
print(" Spatial smoothing...")
|
||||||
|
sector_smoothed = spatial_smooth(sector_shrunk, centroids, sector_n)
|
||||||
|
|
||||||
|
# Collect rows
|
||||||
|
for sec in all_sectors:
|
||||||
|
prem = sector_smoothed.get(sec, national_prem)
|
||||||
|
n = sector_n.get(sec, 0)
|
||||||
|
for et in EVENT_TYPES:
|
||||||
|
rows.append((sec, tg, et, prem.get(et, 0.0), n))
|
||||||
|
|
||||||
|
result = pl.DataFrame(
|
||||||
|
rows,
|
||||||
|
schema={
|
||||||
|
"sector": pl.String,
|
||||||
|
"type_group": pl.String,
|
||||||
|
"event_type": pl.String,
|
||||||
|
"log_premium": pl.Float64,
|
||||||
|
"n_reno_pairs": pl.Int64,
|
||||||
|
},
|
||||||
|
orient="row",
|
||||||
|
).sort("type_group", "sector", "event_type")
|
||||||
|
|
||||||
|
result.write_parquet(args.output)
|
||||||
|
size_mb = args.output.stat().st_size / (1024 * 1024)
|
||||||
|
print(f"\nWrote {args.output} ({size_mb:.1f} MB)")
|
||||||
|
print(
|
||||||
|
f" {result['sector'].n_unique():,} sectors x {len(all_type_groups)} types x {len(EVENT_TYPES)} events = {len(result):,} rows"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Print summary statistics
|
||||||
|
print("\nNational premium summary:")
|
||||||
|
national = (
|
||||||
|
result.filter(pl.col("type_group") == "All")
|
||||||
|
.group_by("event_type")
|
||||||
|
.agg(
|
||||||
|
pl.col("log_premium").mean().alias("mean_log_premium"),
|
||||||
|
)
|
||||||
|
)
|
||||||
|
for row in national.iter_rows(named=True):
|
||||||
|
et = row["event_type"]
|
||||||
|
lp = row["mean_log_premium"]
|
||||||
|
print(f" {et}: log_premium={lp:.4f} ({math.exp(lp) - 1:.1%} price uplift)")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
|
|
@ -163,7 +163,7 @@ public class App {
|
||||||
case "bicycle":
|
case "bicycle":
|
||||||
task.fromTime = 8 * 3600;
|
task.fromTime = 8 * 3600;
|
||||||
task.toTime = 8 * 3600 + 60;
|
task.toTime = 8 * 3600 + 60;
|
||||||
task.maxTripDurationMinutes = 90;
|
task.maxTripDurationMinutes = 120;
|
||||||
task.accessModes = EnumSet.of(LegMode.BICYCLE);
|
task.accessModes = EnumSet.of(LegMode.BICYCLE);
|
||||||
task.egressModes = EnumSet.of(LegMode.BICYCLE);
|
task.egressModes = EnumSet.of(LegMode.BICYCLE);
|
||||||
task.directModes = EnumSet.of(LegMode.BICYCLE);
|
task.directModes = EnumSet.of(LegMode.BICYCLE);
|
||||||
|
|
@ -172,7 +172,7 @@ public class App {
|
||||||
case "walking":
|
case "walking":
|
||||||
task.fromTime = 8 * 3600;
|
task.fromTime = 8 * 3600;
|
||||||
task.toTime = 8 * 3600 + 60;
|
task.toTime = 8 * 3600 + 60;
|
||||||
task.maxTripDurationMinutes = 60;
|
task.maxTripDurationMinutes = 120;
|
||||||
task.accessModes = EnumSet.of(LegMode.WALK);
|
task.accessModes = EnumSet.of(LegMode.WALK);
|
||||||
task.egressModes = EnumSet.of(LegMode.WALK);
|
task.egressModes = EnumSet.of(LegMode.WALK);
|
||||||
task.directModes = EnumSet.of(LegMode.WALK);
|
task.directModes = EnumSet.of(LegMode.WALK);
|
||||||
|
|
@ -181,7 +181,7 @@ public class App {
|
||||||
default: // transit
|
default: // transit
|
||||||
task.fromTime = 8 * 3600;
|
task.fromTime = 8 * 3600;
|
||||||
task.toTime = 8 * 3600 + 60; // single RAPTOR sweep
|
task.toTime = 8 * 3600 + 60; // single RAPTOR sweep
|
||||||
task.maxTripDurationMinutes = 90;
|
task.maxTripDurationMinutes = 120;
|
||||||
task.maxRides = 4;
|
task.maxRides = 4;
|
||||||
task.accessModes = EnumSet.of(LegMode.WALK);
|
task.accessModes = EnumSet.of(LegMode.WALK);
|
||||||
task.egressModes = EnumSet.of(LegMode.WALK);
|
task.egressModes = EnumSet.of(LegMode.WALK);
|
||||||
|
|
|
||||||
|
|
@ -79,13 +79,18 @@ async fn validate_token(
|
||||||
.header("Authorization", format!("Bearer {token}"))
|
.header("Authorization", format!("Bearer {token}"))
|
||||||
.send()
|
.send()
|
||||||
.await
|
.await
|
||||||
|
.map_err(|err| warn!("Token validation request failed: {err}"))
|
||||||
.ok()?;
|
.ok()?;
|
||||||
|
|
||||||
if !res.status().is_success() {
|
if !res.status().is_success() {
|
||||||
return None;
|
return None;
|
||||||
}
|
}
|
||||||
|
|
||||||
let body: AuthRefreshResponse = res.json().await.ok()?;
|
let body: AuthRefreshResponse = res
|
||||||
|
.json()
|
||||||
|
.await
|
||||||
|
.map_err(|err| warn!("Failed to parse auth refresh response: {err}"))
|
||||||
|
.ok()?;
|
||||||
Some(body.record)
|
Some(body.record)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -18,6 +18,5 @@ pub const AREA_SUMMARY_SYSTEM_PROMPT: &str = "You are an experienced estate agen
|
||||||
pub const AREA_SUMMARY_MAX_TOKENS: usize = 300;
|
pub const AREA_SUMMARY_MAX_TOKENS: usize = 300;
|
||||||
pub const AREA_SUMMARY_TEMPERATURE: f32 = 0.3;
|
pub const AREA_SUMMARY_TEMPERATURE: f32 = 0.3;
|
||||||
|
|
||||||
pub const AI_FILTERS_SYSTEM_PROMPT: &str = "You are a property search assistant. The user will describe their ideal property or area in natural language. Your job is to translate their description into filter settings. ONLY set filters the user explicitly mentioned or clearly implied. Leave everything else out. Do not guess or add extra filters. If a request is ambiguous, prefer a wider range. Output valid JSON matching the provided schema.";
|
|
||||||
pub const AI_FILTERS_MAX_TOKENS: usize = 2000;
|
pub const AI_FILTERS_MAX_TOKENS: usize = 2000;
|
||||||
pub const AI_FILTERS_TEMPERATURE: f32 = 0.0;
|
pub const AI_FILTERS_TEMPERATURE: f32 = 0.0;
|
||||||
|
|
|
||||||
|
|
@ -26,6 +26,8 @@ pub struct FeatureConfig {
|
||||||
pub suffix: &'static str,
|
pub suffix: &'static str,
|
||||||
/// If true, show full integer (no k/M abbreviation)
|
/// If true, show full integer (no k/M abbreviation)
|
||||||
pub raw: bool,
|
pub raw: bool,
|
||||||
|
/// If true, the slider uses absolute min/max/step instead of percentile scaling
|
||||||
|
pub absolute: bool,
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Features whose histogram bins should be exactly 1 unit wide (one per integer).
|
/// Features whose histogram bins should be exactly 1 unit wide (one per integer).
|
||||||
|
|
@ -85,6 +87,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "£",
|
prefix: "£",
|
||||||
suffix: "",
|
suffix: "",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: true,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Estimated current price",
|
name: "Estimated current price",
|
||||||
|
|
@ -94,11 +97,12 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
},
|
},
|
||||||
step: 10000.0,
|
step: 10000.0,
|
||||||
description: "Inflation-adjusted estimate of the current property value",
|
description: "Inflation-adjusted estimate of the current property value",
|
||||||
detail: "Estimated by applying a repeat-sales price index to the last known sale price. The index tracks price changes within each postcode sector and property type. Properties sold recently will have estimates close to their sale price; older sales are adjusted more. Coverage depends on having enough repeat sales in the local area to build the index.",
|
detail: "Estimated by applying a repeat-sales price index to the last known sale price, plus a renovation premium for properties with post-sale improvements detected from EPC records (extensions, renovations, remodeling). The index tracks price changes within each postcode sector and property type. Renovation premiums are estimated per area from observed repeat-sale pairs and decay over time. Properties sold recently will have estimates close to their sale price; older sales are adjusted more.",
|
||||||
source: "price-paid",
|
source: "price-paid",
|
||||||
prefix: "£",
|
prefix: "£",
|
||||||
suffix: "",
|
suffix: "",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: true,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Price per sqm",
|
name: "Price per sqm",
|
||||||
|
|
@ -113,6 +117,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "£",
|
prefix: "£",
|
||||||
suffix: "",
|
suffix: "",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Est. price per sqm",
|
name: "Est. price per sqm",
|
||||||
|
|
@ -122,11 +127,12 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
},
|
},
|
||||||
step: 100.0,
|
step: 100.0,
|
||||||
description: "Estimated current price divided by total floor area",
|
description: "Estimated current price divided by total floor area",
|
||||||
detail: "Calculated by dividing the inflation-adjusted estimated current price by the total floor area from the EPC certificate. Provides a more up-to-date price-per-area comparison than the historical sale price per sqm.",
|
detail: "Calculated by dividing the inflation-adjusted estimated current price (including any renovation premium) by the total floor area from the EPC certificate. Provides a more up-to-date price-per-area comparison than the historical sale price per sqm.",
|
||||||
source: "price-paid",
|
source: "price-paid",
|
||||||
prefix: "£",
|
prefix: "£",
|
||||||
suffix: "",
|
suffix: "",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Total floor area (sqm)",
|
name: "Total floor area (sqm)",
|
||||||
|
|
@ -141,12 +147,28 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: " sqm",
|
suffix: " sqm",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
|
},
|
||||||
|
FeatureConfig {
|
||||||
|
name: "Interior height (m)",
|
||||||
|
bounds: Bounds::Percentile {
|
||||||
|
low: 2.0,
|
||||||
|
high: 98.0,
|
||||||
|
},
|
||||||
|
step: 0.1,
|
||||||
|
description: "Average storey height from the EPC survey",
|
||||||
|
detail: "Average internal floor-to-ceiling height in metres as recorded during the Energy Performance Certificate assessment. Calculated by dividing the total internal volume by the total floor area.",
|
||||||
|
source: "epc",
|
||||||
|
prefix: "",
|
||||||
|
suffix: " m",
|
||||||
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Number of bedrooms & living rooms",
|
name: "Number of bedrooms & living rooms",
|
||||||
bounds: Bounds::Fixed {
|
bounds: Bounds::Fixed {
|
||||||
min: 1.0,
|
min: 1.0,
|
||||||
max: 10.0,
|
max: 12.0,
|
||||||
},
|
},
|
||||||
step: 1.0,
|
step: 1.0,
|
||||||
description: "Count of habitable rooms from the EPC survey",
|
description: "Count of habitable rooms from the EPC survey",
|
||||||
|
|
@ -155,6 +177,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: " rooms",
|
suffix: " rooms",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: true,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Estimated monthly rent",
|
name: "Estimated monthly rent",
|
||||||
|
|
@ -166,6 +189,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "£",
|
prefix: "£",
|
||||||
suffix: "/mo",
|
suffix: "/mo",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Date of last transaction",
|
name: "Date of last transaction",
|
||||||
|
|
@ -180,6 +204,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "",
|
suffix: "",
|
||||||
raw: true,
|
raw: true,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Construction age",
|
name: "Construction age",
|
||||||
|
|
@ -194,6 +219,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "",
|
suffix: "",
|
||||||
raw: true,
|
raw: true,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
|
|
@ -213,6 +239,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: " mins",
|
suffix: " mins",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Public transport to Fitzrovia (mins)",
|
name: "Public transport to Fitzrovia (mins)",
|
||||||
|
|
@ -227,6 +254,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: " mins",
|
suffix: " mins",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Cycling to Bank (mins)",
|
name: "Cycling to Bank (mins)",
|
||||||
|
|
@ -241,6 +269,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: " mins",
|
suffix: " mins",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Cycling to Fitzrovia (mins)",
|
name: "Cycling to Fitzrovia (mins)",
|
||||||
|
|
@ -255,6 +284,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: " mins",
|
suffix: " mins",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Number of public transport stations within 2km",
|
name: "Number of public transport stations within 2km",
|
||||||
|
|
@ -269,6 +299,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "",
|
suffix: "",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
|
|
@ -288,6 +319,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "",
|
suffix: "",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Good+ primary schools within 5km",
|
name: "Good+ primary schools within 5km",
|
||||||
|
|
@ -302,6 +334,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "",
|
suffix: "",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Good+ secondary schools within 5km",
|
name: "Good+ secondary schools within 5km",
|
||||||
|
|
@ -316,6 +349,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "",
|
suffix: "",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
|
|
@ -332,6 +366,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "",
|
suffix: "",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Employment Score (rate)",
|
name: "Employment Score (rate)",
|
||||||
|
|
@ -343,6 +378,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "",
|
suffix: "",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Health Deprivation and Disability Score",
|
name: "Health Deprivation and Disability Score",
|
||||||
|
|
@ -357,6 +393,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "",
|
suffix: "",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Living Environment Score",
|
name: "Living Environment Score",
|
||||||
|
|
@ -371,6 +408,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "",
|
suffix: "",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Indoors Sub-domain Score",
|
name: "Indoors Sub-domain Score",
|
||||||
|
|
@ -385,6 +423,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "",
|
suffix: "",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Outdoors Sub-domain Score",
|
name: "Outdoors Sub-domain Score",
|
||||||
|
|
@ -399,6 +438,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "",
|
suffix: "",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
|
|
@ -418,6 +458,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "/yr",
|
suffix: "/yr",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Violence and sexual offences (avg/yr)",
|
name: "Violence and sexual offences (avg/yr)",
|
||||||
|
|
@ -432,6 +473,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "/yr",
|
suffix: "/yr",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Criminal damage and arson (avg/yr)",
|
name: "Criminal damage and arson (avg/yr)",
|
||||||
|
|
@ -446,6 +488,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "/yr",
|
suffix: "/yr",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Burglary (avg/yr)",
|
name: "Burglary (avg/yr)",
|
||||||
|
|
@ -460,6 +503,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "/yr",
|
suffix: "/yr",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Vehicle crime (avg/yr)",
|
name: "Vehicle crime (avg/yr)",
|
||||||
|
|
@ -474,6 +518,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "/yr",
|
suffix: "/yr",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Robbery (avg/yr)",
|
name: "Robbery (avg/yr)",
|
||||||
|
|
@ -488,6 +533,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "/yr",
|
suffix: "/yr",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Other theft (avg/yr)",
|
name: "Other theft (avg/yr)",
|
||||||
|
|
@ -502,6 +548,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "/yr",
|
suffix: "/yr",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Shoplifting (avg/yr)",
|
name: "Shoplifting (avg/yr)",
|
||||||
|
|
@ -516,6 +563,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "/yr",
|
suffix: "/yr",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Drugs (avg/yr)",
|
name: "Drugs (avg/yr)",
|
||||||
|
|
@ -530,6 +578,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "/yr",
|
suffix: "/yr",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Possession of weapons (avg/yr)",
|
name: "Possession of weapons (avg/yr)",
|
||||||
|
|
@ -544,6 +593,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "/yr",
|
suffix: "/yr",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Public order (avg/yr)",
|
name: "Public order (avg/yr)",
|
||||||
|
|
@ -558,6 +608,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "/yr",
|
suffix: "/yr",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Bicycle theft (avg/yr)",
|
name: "Bicycle theft (avg/yr)",
|
||||||
|
|
@ -572,6 +623,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "/yr",
|
suffix: "/yr",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Theft from the person (avg/yr)",
|
name: "Theft from the person (avg/yr)",
|
||||||
|
|
@ -586,6 +638,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "/yr",
|
suffix: "/yr",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Other crime (avg/yr)",
|
name: "Other crime (avg/yr)",
|
||||||
|
|
@ -600,6 +653,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "/yr",
|
suffix: "/yr",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Serious crime (avg/yr)",
|
name: "Serious crime (avg/yr)",
|
||||||
|
|
@ -614,6 +668,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "/yr",
|
suffix: "/yr",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Minor crime (avg/yr)",
|
name: "Minor crime (avg/yr)",
|
||||||
|
|
@ -628,6 +683,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "/yr",
|
suffix: "/yr",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
|
|
@ -647,6 +703,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "%",
|
suffix: "%",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "% Asian",
|
name: "% Asian",
|
||||||
|
|
@ -661,6 +718,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "%",
|
suffix: "%",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "% Black",
|
name: "% Black",
|
||||||
|
|
@ -675,6 +733,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "%",
|
suffix: "%",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "% Mixed",
|
name: "% Mixed",
|
||||||
|
|
@ -689,6 +748,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "%",
|
suffix: "%",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "% Other",
|
name: "% Other",
|
||||||
|
|
@ -703,6 +763,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "%",
|
suffix: "%",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
|
|
@ -722,6 +783,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "",
|
suffix: "",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Number of grocery shops and supermarkets within 2km",
|
name: "Number of grocery shops and supermarkets within 2km",
|
||||||
|
|
@ -736,6 +798,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "",
|
suffix: "",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Number of parks within 2km",
|
name: "Number of parks within 2km",
|
||||||
|
|
@ -750,6 +813,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: "",
|
suffix: "",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
|
|
@ -769,6 +833,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: " dB",
|
suffix: " dB",
|
||||||
raw: false,
|
raw: false,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
FeatureConfig {
|
FeatureConfig {
|
||||||
name: "Max available download speed (Mbps)",
|
name: "Max available download speed (Mbps)",
|
||||||
|
|
@ -783,6 +848,7 @@ pub static FEATURE_GROUPS: &[FeatureGroup] = &[
|
||||||
prefix: "",
|
prefix: "",
|
||||||
suffix: " Mbps",
|
suffix: " Mbps",
|
||||||
raw: true,
|
raw: true,
|
||||||
|
absolute: false,
|
||||||
},
|
},
|
||||||
],
|
],
|
||||||
},
|
},
|
||||||
|
|
|
||||||
|
|
@ -6,6 +6,7 @@ mod features;
|
||||||
mod metrics;
|
mod metrics;
|
||||||
mod og_middleware;
|
mod og_middleware;
|
||||||
pub mod parsing;
|
pub mod parsing;
|
||||||
|
mod pocketbase;
|
||||||
mod routes;
|
mod routes;
|
||||||
mod state;
|
mod state;
|
||||||
pub mod utils;
|
pub mod utils;
|
||||||
|
|
@ -23,7 +24,7 @@ use tower_http::compression::CompressionLayer;
|
||||||
use tower_http::cors::{Any, CorsLayer};
|
use tower_http::cors::{Any, CorsLayer};
|
||||||
use tower_http::services::{ServeDir, ServeFile};
|
use tower_http::services::{ServeDir, ServeFile};
|
||||||
use tower_http::trace::TraceLayer;
|
use tower_http::trace::TraceLayer;
|
||||||
use tracing::{info, warn};
|
use tracing::info;
|
||||||
use tracing_subscriber::EnvFilter;
|
use tracing_subscriber::EnvFilter;
|
||||||
|
|
||||||
use state::AppState;
|
use state::AppState;
|
||||||
|
|
@ -39,6 +40,10 @@ struct Cli {
|
||||||
#[arg(long)]
|
#[arg(long)]
|
||||||
pois: PathBuf,
|
pois: PathBuf,
|
||||||
|
|
||||||
|
/// Path to the places parquet file
|
||||||
|
#[arg(long)]
|
||||||
|
places: PathBuf,
|
||||||
|
|
||||||
/// Path to the postcode boundaries directory
|
/// Path to the postcode boundaries directory
|
||||||
#[arg(long)]
|
#[arg(long)]
|
||||||
postcodes: PathBuf,
|
postcodes: PathBuf,
|
||||||
|
|
@ -56,28 +61,36 @@ struct Cli {
|
||||||
screenshot_url: String,
|
screenshot_url: String,
|
||||||
|
|
||||||
/// Public-facing URL for absolute og:image URLs
|
/// Public-facing URL for absolute og:image URLs
|
||||||
#[arg(
|
#[arg(long, env = "PUBLIC_URL")]
|
||||||
long,
|
|
||||||
env = "PUBLIC_URL",
|
|
||||||
default_value = "https://perfectpostcodes.schmelczer.dev"
|
|
||||||
)]
|
|
||||||
public_url: String,
|
public_url: String,
|
||||||
|
|
||||||
/// PocketBase server URL for authentication (e.g. http://localhost:8090)
|
/// PocketBase server URL for authentication (e.g. http://localhost:8090)
|
||||||
#[arg(long, env = "POCKETBASE_URL")]
|
#[arg(long, env = "POCKETBASE_URL")]
|
||||||
pocketbase_url: String,
|
pocketbase_url: String,
|
||||||
|
|
||||||
|
/// PocketBase superuser email (for auto-creating collections at startup)
|
||||||
|
#[arg(long, env = "POCKETBASE_ADMIN_EMAIL")]
|
||||||
|
pocketbase_admin_email: Option<String>,
|
||||||
|
|
||||||
|
/// PocketBase superuser password (for auto-creating collections at startup)
|
||||||
|
#[arg(long, env = "POCKETBASE_ADMIN_PASSWORD")]
|
||||||
|
pocketbase_admin_password: Option<String>,
|
||||||
|
|
||||||
/// Ollama server URL for AI area summaries (e.g. http://ollama:11434)
|
/// Ollama server URL for AI area summaries (e.g. http://ollama:11434)
|
||||||
#[arg(long, env = "OLLAMA_URL")]
|
#[arg(long, env = "OLLAMA_URL")]
|
||||||
ollama_url: String,
|
ollama_url: String,
|
||||||
|
|
||||||
/// Ollama model name for area summaries
|
/// Ollama model name for area summaries
|
||||||
#[arg(long, env = "OLLAMA_MODEL", default_value = "gemma3:12b")]
|
#[arg(long, env = "OLLAMA_MODEL")]
|
||||||
ollama_model: String,
|
ollama_model: String,
|
||||||
|
|
||||||
/// R5 routing service URL for real-time travel times (e.g. http://r5:8003)
|
/// R5 routing service URL for all travel times (e.g. http://r5:8003)
|
||||||
#[arg(long, env = "R5_URL", default_value = "")]
|
#[arg(long, env = "R5_URL")]
|
||||||
r5_url: String,
|
r5_url: Option<String>,
|
||||||
|
|
||||||
|
/// Google Maps API key for Street View metadata lookups
|
||||||
|
#[arg(long, env = "GOOGLE_MAPS_API_KEY")]
|
||||||
|
google_maps_api_key: String,
|
||||||
}
|
}
|
||||||
|
|
||||||
#[tokio::main]
|
#[tokio::main]
|
||||||
|
|
@ -138,6 +151,15 @@ async fn main() -> anyhow::Result<()> {
|
||||||
info!("Building POI spatial grid index");
|
info!("Building POI spatial grid index");
|
||||||
let poi_grid = utils::GridIndex::build(&poi_data.lat, &poi_data.lng, consts::GRID_CELL_SIZE);
|
let poi_grid = utils::GridIndex::build(&poi_data.lat, &poi_data.lng, consts::GRID_CELL_SIZE);
|
||||||
|
|
||||||
|
// Load place data
|
||||||
|
let places_path = &cli.places;
|
||||||
|
if !places_path.exists() {
|
||||||
|
bail!("Places parquet file not found: {}", places_path.display());
|
||||||
|
}
|
||||||
|
info!("Loading place data from {}", places_path.display());
|
||||||
|
let place_data = data::PlaceData::load(places_path)?;
|
||||||
|
info!(places = place_data.name.len(), "Place data loaded");
|
||||||
|
|
||||||
// Load postcode boundaries
|
// Load postcode boundaries
|
||||||
let postcodes_path = &cli.postcodes;
|
let postcodes_path = &cli.postcodes;
|
||||||
if !postcodes_path.exists() {
|
if !postcodes_path.exists() {
|
||||||
|
|
@ -191,26 +213,15 @@ async fn main() -> anyhow::Result<()> {
|
||||||
let poi_category_groups = poi_data.category_groups()?;
|
let poi_category_groups = poi_data.category_groups()?;
|
||||||
|
|
||||||
// Read index.html at startup for crawler OG injection
|
// Read index.html at startup for crawler OG injection
|
||||||
let frontend_dist = cli
|
let (frontend_dist, index_html) = if let Some(dist) = cli.dist {
|
||||||
.dist
|
let index_path = dist.join("index.html");
|
||||||
.unwrap_or_else(|| PathBuf::from("frontend/dist"));
|
let html = std::fs::read_to_string(&index_path)
|
||||||
|
.with_context(|| format!("Failed to read {}", index_path.display()))?;
|
||||||
let index_html = {
|
info!("Loaded index.html for OG injection");
|
||||||
let index_path = frontend_dist.join("index.html");
|
(Some(dist), Some(html))
|
||||||
match std::fs::read_to_string(&index_path) {
|
} else {
|
||||||
Ok(html) => {
|
info!("No --dist provided, static serving and OG injection disabled");
|
||||||
info!("Loaded index.html for OG injection");
|
(None, None)
|
||||||
Some(html)
|
|
||||||
}
|
|
||||||
Err(err) => {
|
|
||||||
warn!(
|
|
||||||
"Could not read {}: {} (OG injection disabled)",
|
|
||||||
index_path.display(),
|
|
||||||
err
|
|
||||||
);
|
|
||||||
None
|
|
||||||
}
|
|
||||||
}
|
|
||||||
};
|
};
|
||||||
|
|
||||||
let http_client = reqwest::Client::new();
|
let http_client = reqwest::Client::new();
|
||||||
|
|
@ -223,6 +234,10 @@ async fn main() -> anyhow::Result<()> {
|
||||||
"Precomputed features response"
|
"Precomputed features response"
|
||||||
);
|
);
|
||||||
|
|
||||||
|
let ai_filters_schema = routes::build_ollama_schema(&features_response);
|
||||||
|
let ai_filters_system_prompt = routes::build_system_prompt(&features_response);
|
||||||
|
info!("Precomputed AI filters schema and system prompt");
|
||||||
|
|
||||||
// Record data loading metrics
|
// Record data loading metrics
|
||||||
metrics::record_data_stats(
|
metrics::record_data_stats(
|
||||||
property_data.lat.len(),
|
property_data.lat.len(),
|
||||||
|
|
@ -231,12 +246,21 @@ async fn main() -> anyhow::Result<()> {
|
||||||
);
|
);
|
||||||
|
|
||||||
info!("PocketBase configured: {}", cli.pocketbase_url);
|
info!("PocketBase configured: {}", cli.pocketbase_url);
|
||||||
|
|
||||||
|
if let (Some(ref email), Some(ref password)) =
|
||||||
|
(&cli.pocketbase_admin_email, &cli.pocketbase_admin_password)
|
||||||
|
{
|
||||||
|
pocketbase::ensure_collections(&http_client, &cli.pocketbase_url, email, password).await?;
|
||||||
|
} else {
|
||||||
|
info!("PocketBase admin credentials not set — skipping collection auto-creation");
|
||||||
|
}
|
||||||
|
|
||||||
info!(
|
info!(
|
||||||
"Ollama configured: {} (model: {})",
|
"Ollama configured: {} (model: {})",
|
||||||
cli.ollama_url, cli.ollama_model
|
cli.ollama_url, cli.ollama_model
|
||||||
);
|
);
|
||||||
if !cli.r5_url.is_empty() {
|
if let Some(ref url) = cli.r5_url {
|
||||||
info!("R5 routing service configured: {}", cli.r5_url);
|
info!("R5 routing service configured: {}", url);
|
||||||
} else {
|
} else {
|
||||||
info!("R5 routing service not configured (travel time queries disabled)");
|
info!("R5 routing service not configured (travel time queries disabled)");
|
||||||
}
|
}
|
||||||
|
|
@ -249,6 +273,7 @@ async fn main() -> anyhow::Result<()> {
|
||||||
h3_cells,
|
h3_cells,
|
||||||
poi_data,
|
poi_data,
|
||||||
poi_grid,
|
poi_grid,
|
||||||
|
place_data,
|
||||||
postcode_data,
|
postcode_data,
|
||||||
feature_name_to_index,
|
feature_name_to_index,
|
||||||
min_keys,
|
min_keys,
|
||||||
|
|
@ -265,6 +290,9 @@ async fn main() -> anyhow::Result<()> {
|
||||||
ollama_model: cli.ollama_model,
|
ollama_model: cli.ollama_model,
|
||||||
r5_url: cli.r5_url,
|
r5_url: cli.r5_url,
|
||||||
token_cache,
|
token_cache,
|
||||||
|
ai_filters_schema,
|
||||||
|
ai_filters_system_prompt,
|
||||||
|
google_maps_api_key: cli.google_maps_api_key,
|
||||||
});
|
});
|
||||||
|
|
||||||
let cors = CorsLayer::new()
|
let cors = CorsLayer::new()
|
||||||
|
|
@ -286,8 +314,11 @@ async fn main() -> anyhow::Result<()> {
|
||||||
let state_pb = state.clone();
|
let state_pb = state.clone();
|
||||||
let state_postcode_stats = state.clone();
|
let state_postcode_stats = state.clone();
|
||||||
let state_area_summary = state.clone();
|
let state_area_summary = state.clone();
|
||||||
|
let state_places = state.clone();
|
||||||
let state_shorten = state.clone();
|
let state_shorten = state.clone();
|
||||||
let state_short_url = state.clone();
|
let state_short_url = state.clone();
|
||||||
|
let state_ai_filters = state.clone();
|
||||||
|
let state_streetview = state.clone();
|
||||||
|
|
||||||
let api = Router::new()
|
let api = Router::new()
|
||||||
.route(
|
.route(
|
||||||
|
|
@ -314,6 +345,10 @@ async fn main() -> anyhow::Result<()> {
|
||||||
"/api/poi-categories",
|
"/api/poi-categories",
|
||||||
get(move || routes::get_poi_categories(state_poi_categories.clone())),
|
get(move || routes::get_poi_categories(state_poi_categories.clone())),
|
||||||
)
|
)
|
||||||
|
.route(
|
||||||
|
"/api/places",
|
||||||
|
get(move |query| routes::get_places(state_places.clone(), query)),
|
||||||
|
)
|
||||||
.route(
|
.route(
|
||||||
"/api/hexagon-properties",
|
"/api/hexagon-properties",
|
||||||
get(move |query| {
|
get(move |query| {
|
||||||
|
|
@ -345,6 +380,14 @@ async fn main() -> anyhow::Result<()> {
|
||||||
"/api/shorten",
|
"/api/shorten",
|
||||||
post(move |body| routes::post_shorten(state_shorten.clone(), body)),
|
post(move |body| routes::post_shorten(state_shorten.clone(), body)),
|
||||||
)
|
)
|
||||||
|
.route(
|
||||||
|
"/api/ai-filters",
|
||||||
|
post(move |body| routes::post_ai_filters(state_ai_filters.clone(), body)),
|
||||||
|
)
|
||||||
|
.route(
|
||||||
|
"/api/streetview",
|
||||||
|
get(move |query| routes::get_streetview(state_streetview.clone(), query)),
|
||||||
|
)
|
||||||
.route(
|
.route(
|
||||||
"/s/{code}",
|
"/s/{code}",
|
||||||
get(move |path| routes::get_short_url(state_short_url.clone(), path)),
|
get(move |path| routes::get_short_url(state_short_url.clone(), path)),
|
||||||
|
|
@ -364,6 +407,7 @@ async fn main() -> anyhow::Result<()> {
|
||||||
routes::get_style(axum::extract::State(reader_style.clone()), headers, query)
|
routes::get_style(axum::extract::State(reader_style.clone()), headers, query)
|
||||||
}),
|
}),
|
||||||
)
|
)
|
||||||
|
.route("/health", get(|| async { "ok" }))
|
||||||
.route(
|
.route(
|
||||||
"/metrics",
|
"/metrics",
|
||||||
get(move || metrics::metrics_handler(metrics_handle.clone())),
|
get(move || metrics::metrics_handler(metrics_handle.clone())),
|
||||||
|
|
@ -373,10 +417,9 @@ async fn main() -> anyhow::Result<()> {
|
||||||
any(move |req| routes::proxy_to_pocketbase(state_pb.clone(), req)),
|
any(move |req| routes::proxy_to_pocketbase(state_pb.clone(), req)),
|
||||||
);
|
);
|
||||||
|
|
||||||
let app = if frontend_dist.exists() {
|
let app = if let Some(ref dist) = frontend_dist {
|
||||||
api.fallback_service(
|
api.fallback_service(
|
||||||
ServeDir::new(&frontend_dist)
|
ServeDir::new(dist).not_found_service(ServeFile::new(dist.join("index.html"))),
|
||||||
.not_found_service(ServeFile::new(frontend_dist.join("index.html"))),
|
|
||||||
)
|
)
|
||||||
} else {
|
} else {
|
||||||
api
|
api
|
||||||
|
|
|
||||||
|
|
@ -38,11 +38,8 @@ pub fn cell_for_row(
|
||||||
if !need_parent || max_cell == 0 {
|
if !need_parent || max_cell == 0 {
|
||||||
return max_cell;
|
return max_cell;
|
||||||
}
|
}
|
||||||
h3o::CellIndex::try_from(max_cell)
|
let cell = h3o::CellIndex::try_from(max_cell).expect("precomputed H3 cell must be valid");
|
||||||
.ok()
|
u64::from(cell.parent(h3_res).expect("parent resolution must be valid for precomputed cell"))
|
||||||
.and_then(|ci| ci.parent(h3_res))
|
|
||||||
.map(u64::from)
|
|
||||||
.unwrap_or(0)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Whether the given resolution requires computing a parent from precomputed cells.
|
/// Whether the given resolution requires computing a parent from precomputed cells.
|
||||||
|
|
|
||||||
235
server-rs/src/pocketbase.rs
Normal file
235
server-rs/src/pocketbase.rs
Normal file
|
|
@ -0,0 +1,235 @@
|
||||||
|
use reqwest::Client;
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
use tracing::info;
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
struct AuthResponse {
|
||||||
|
token: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
struct CollectionList {
|
||||||
|
items: Vec<CollectionItem>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
struct CollectionItem {
|
||||||
|
name: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Serialize)]
|
||||||
|
struct CreateCollection {
|
||||||
|
name: String,
|
||||||
|
r#type: String,
|
||||||
|
fields: Vec<Field>,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Serialize)]
|
||||||
|
#[serde(rename_all = "camelCase")]
|
||||||
|
struct Field {
|
||||||
|
name: String,
|
||||||
|
r#type: String,
|
||||||
|
#[serde(skip_serializing_if = "Option::is_none")]
|
||||||
|
required: Option<bool>,
|
||||||
|
#[serde(skip_serializing_if = "Option::is_none")]
|
||||||
|
max_select: Option<u32>,
|
||||||
|
#[serde(skip_serializing_if = "Option::is_none")]
|
||||||
|
collection_id: Option<String>,
|
||||||
|
#[serde(skip_serializing_if = "Option::is_none")]
|
||||||
|
max_size: Option<u64>,
|
||||||
|
#[serde(skip_serializing_if = "Option::is_none")]
|
||||||
|
mime_types: Option<Vec<String>>,
|
||||||
|
}
|
||||||
|
|
||||||
|
impl Field {
|
||||||
|
fn text(name: &str, required: bool) -> Self {
|
||||||
|
Self {
|
||||||
|
name: name.to_string(),
|
||||||
|
r#type: "text".to_string(),
|
||||||
|
required: Some(required),
|
||||||
|
max_select: None,
|
||||||
|
collection_id: None,
|
||||||
|
max_size: None,
|
||||||
|
mime_types: None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn file(name: &str, mime_types: Vec<&str>) -> Self {
|
||||||
|
Self {
|
||||||
|
name: name.to_string(),
|
||||||
|
r#type: "file".to_string(),
|
||||||
|
required: Some(false),
|
||||||
|
max_select: Some(1),
|
||||||
|
collection_id: None,
|
||||||
|
max_size: Some(10 * 1024 * 1024), // 10 MB
|
||||||
|
mime_types: Some(mime_types.into_iter().map(String::from).collect()),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fn relation(name: &str, collection_id: &str) -> Self {
|
||||||
|
Self {
|
||||||
|
name: name.to_string(),
|
||||||
|
r#type: "relation".to_string(),
|
||||||
|
required: Some(true),
|
||||||
|
max_select: Some(1),
|
||||||
|
collection_id: Some(collection_id.to_string()),
|
||||||
|
max_size: None,
|
||||||
|
mime_types: None,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn auth_superuser(
|
||||||
|
client: &Client,
|
||||||
|
base_url: &str,
|
||||||
|
email: &str,
|
||||||
|
password: &str,
|
||||||
|
) -> anyhow::Result<String> {
|
||||||
|
let url = format!("{base_url}/api/collections/_superusers/auth-with-password");
|
||||||
|
let resp = client
|
||||||
|
.post(&url)
|
||||||
|
.json(&serde_json::json!({
|
||||||
|
"identity": email,
|
||||||
|
"password": password,
|
||||||
|
}))
|
||||||
|
.send()
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
if !resp.status().is_success() {
|
||||||
|
let status = resp.status();
|
||||||
|
let text = resp.text().await.unwrap_or_default();
|
||||||
|
anyhow::bail!("PocketBase superuser auth failed ({status}): {text}");
|
||||||
|
}
|
||||||
|
|
||||||
|
let body: AuthResponse = resp.json().await?;
|
||||||
|
Ok(body.token)
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn list_collections(
|
||||||
|
client: &Client,
|
||||||
|
base_url: &str,
|
||||||
|
token: &str,
|
||||||
|
) -> anyhow::Result<Vec<String>> {
|
||||||
|
let url = format!("{base_url}/api/collections?perPage=200");
|
||||||
|
let resp = client
|
||||||
|
.get(&url)
|
||||||
|
.header("Authorization", format!("Bearer {token}"))
|
||||||
|
.send()
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
if !resp.status().is_success() {
|
||||||
|
let status = resp.status();
|
||||||
|
let text = resp.text().await.unwrap_or_default();
|
||||||
|
anyhow::bail!("Failed to list PocketBase collections ({status}): {text}");
|
||||||
|
}
|
||||||
|
|
||||||
|
let body: CollectionList = resp.json().await?;
|
||||||
|
Ok(body.items.into_iter().map(|c| c.name).collect())
|
||||||
|
}
|
||||||
|
|
||||||
|
async fn create_collection(
|
||||||
|
client: &Client,
|
||||||
|
base_url: &str,
|
||||||
|
token: &str,
|
||||||
|
collection: CreateCollection,
|
||||||
|
) -> anyhow::Result<()> {
|
||||||
|
let name = collection.name.clone();
|
||||||
|
let resp = client
|
||||||
|
.post(&format!("{base_url}/api/collections"))
|
||||||
|
.header("Authorization", format!("Bearer {token}"))
|
||||||
|
.json(&collection)
|
||||||
|
.send()
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
if !resp.status().is_success() {
|
||||||
|
let status = resp.status();
|
||||||
|
let text = resp.text().await.unwrap_or_default();
|
||||||
|
anyhow::bail!("Failed to create collection '{name}' ({status}): {text}");
|
||||||
|
}
|
||||||
|
|
||||||
|
info!("Created PocketBase collection: {name}");
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Look up the internal ID of the `users` auth collection.
|
||||||
|
async fn find_users_collection_id(
|
||||||
|
client: &Client,
|
||||||
|
base_url: &str,
|
||||||
|
token: &str,
|
||||||
|
) -> anyhow::Result<String> {
|
||||||
|
let url = format!("{base_url}/api/collections/users");
|
||||||
|
let resp = client
|
||||||
|
.get(&url)
|
||||||
|
.header("Authorization", format!("Bearer {token}"))
|
||||||
|
.send()
|
||||||
|
.await?;
|
||||||
|
|
||||||
|
if !resp.status().is_success() {
|
||||||
|
let status = resp.status();
|
||||||
|
let text = resp.text().await.unwrap_or_default();
|
||||||
|
anyhow::bail!("Failed to fetch users collection ({status}): {text}");
|
||||||
|
}
|
||||||
|
|
||||||
|
let body: serde_json::Value = resp.json().await?;
|
||||||
|
let id = body["id"]
|
||||||
|
.as_str()
|
||||||
|
.ok_or_else(|| anyhow::anyhow!("users collection has no id field"))?;
|
||||||
|
Ok(id.to_string())
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Ensure the `saved_searches` and `short_urls` collections exist in PocketBase.
|
||||||
|
/// Authenticates as superuser, checks existing collections, and creates any that are missing.
|
||||||
|
pub async fn ensure_collections(
|
||||||
|
client: &Client,
|
||||||
|
base_url: &str,
|
||||||
|
admin_email: &str,
|
||||||
|
admin_password: &str,
|
||||||
|
) -> anyhow::Result<()> {
|
||||||
|
let base_url = base_url.trim_end_matches('/');
|
||||||
|
|
||||||
|
let token = auth_superuser(client, base_url, admin_email, admin_password).await?;
|
||||||
|
let existing = list_collections(client, base_url, &token).await?;
|
||||||
|
|
||||||
|
if !existing.iter().any(|n| n == "saved_searches") {
|
||||||
|
let users_id = find_users_collection_id(client, base_url, &token).await?;
|
||||||
|
create_collection(
|
||||||
|
client,
|
||||||
|
base_url,
|
||||||
|
&token,
|
||||||
|
CreateCollection {
|
||||||
|
name: "saved_searches".to_string(),
|
||||||
|
r#type: "base".to_string(),
|
||||||
|
fields: vec![
|
||||||
|
Field::relation("user", &users_id),
|
||||||
|
Field::text("name", true),
|
||||||
|
Field::text("params", true),
|
||||||
|
Field::file("screenshot", vec!["image/png", "image/jpeg", "image/webp"]),
|
||||||
|
],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
.await?;
|
||||||
|
} else {
|
||||||
|
info!("PocketBase collection 'saved_searches' already exists");
|
||||||
|
}
|
||||||
|
|
||||||
|
if !existing.iter().any(|n| n == "short_urls") {
|
||||||
|
create_collection(
|
||||||
|
client,
|
||||||
|
base_url,
|
||||||
|
&token,
|
||||||
|
CreateCollection {
|
||||||
|
name: "short_urls".to_string(),
|
||||||
|
r#type: "base".to_string(),
|
||||||
|
fields: vec![
|
||||||
|
Field::text("code", true),
|
||||||
|
Field::text("params", true),
|
||||||
|
],
|
||||||
|
},
|
||||||
|
)
|
||||||
|
.await?;
|
||||||
|
} else {
|
||||||
|
info!("PocketBase collection 'short_urls' already exists");
|
||||||
|
}
|
||||||
|
|
||||||
|
Ok(())
|
||||||
|
}
|
||||||
|
|
@ -14,10 +14,11 @@ pub(crate) mod properties;
|
||||||
mod screenshot;
|
mod screenshot;
|
||||||
mod shorten;
|
mod shorten;
|
||||||
mod stats;
|
mod stats;
|
||||||
|
mod streetview;
|
||||||
mod tiles;
|
mod tiles;
|
||||||
pub(crate) mod travel_time;
|
pub(crate) mod travel_time;
|
||||||
|
|
||||||
pub use ai_filters::{build_feature_prompt, build_ollama_schema, post_ai_filters};
|
pub use ai_filters::{build_ollama_schema, build_system_prompt, post_ai_filters};
|
||||||
pub use area_summary::post_area_summary;
|
pub use area_summary::post_area_summary;
|
||||||
pub use export::get_export;
|
pub use export::get_export;
|
||||||
pub use features::{build_features_response, get_features, FeatureInfo, FeaturesResponse};
|
pub use features::{build_features_response, get_features, FeatureInfo, FeaturesResponse};
|
||||||
|
|
@ -32,4 +33,5 @@ pub use postcodes::{get_postcode_lookup, get_postcodes};
|
||||||
pub use properties::get_hexagon_properties;
|
pub use properties::get_hexagon_properties;
|
||||||
pub use screenshot::get_screenshot;
|
pub use screenshot::get_screenshot;
|
||||||
pub use shorten::{get_short_url, post_shorten};
|
pub use shorten::{get_short_url, post_shorten};
|
||||||
|
pub use streetview::get_streetview;
|
||||||
pub use tiles::{get_style, get_tile, init_tile_reader};
|
pub use tiles::{get_style, get_tile, init_tile_reader};
|
||||||
|
|
|
||||||
334
server-rs/src/routes/ai_filters.rs
Normal file
334
server-rs/src/routes/ai_filters.rs
Normal file
|
|
@ -0,0 +1,334 @@
|
||||||
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
use axum::http::StatusCode;
|
||||||
|
use axum::response::Json;
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
use serde_json::{json, Value};
|
||||||
|
use tracing::{info, warn};
|
||||||
|
|
||||||
|
use crate::consts::{AI_FILTERS_MAX_TOKENS, AI_FILTERS_TEMPERATURE};
|
||||||
|
use crate::routes::{FeatureInfo, FeaturesResponse};
|
||||||
|
use crate::state::AppState;
|
||||||
|
use crate::utils::{extract_ollama_content, ollama_chat, strip_think_blocks};
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
pub struct AiFiltersRequest {
|
||||||
|
query: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Serialize)]
|
||||||
|
pub struct AiFiltersResponse {
|
||||||
|
filters: Value,
|
||||||
|
/// What the LLM couldn't map to existing filters (empty if everything matched)
|
||||||
|
#[serde(skip_serializing_if = "String::is_empty")]
|
||||||
|
notes: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Build a JSON schema for Ollama structured output.
|
||||||
|
///
|
||||||
|
/// Uses two arrays (`numeric_filters` and `enum_filters`) instead of one property
|
||||||
|
/// per feature, because Ollama converts JSON schema to GBNF grammar and a schema
|
||||||
|
/// with 50+ optional keys causes a combinatorial explosion that crashes the parser.
|
||||||
|
/// Array-based schema keeps the grammar small and constant-size.
|
||||||
|
pub fn build_ollama_schema(_features: &FeaturesResponse) -> Value {
|
||||||
|
json!({
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"numeric_filters": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"name": { "type": "string" },
|
||||||
|
"min": { "type": "number" },
|
||||||
|
"max": { "type": "number" }
|
||||||
|
},
|
||||||
|
"required": ["name"]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"enum_filters": {
|
||||||
|
"type": "array",
|
||||||
|
"items": {
|
||||||
|
"type": "object",
|
||||||
|
"properties": {
|
||||||
|
"name": { "type": "string" },
|
||||||
|
"values": { "type": "array", "items": { "type": "string" } }
|
||||||
|
},
|
||||||
|
"required": ["name", "values"]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"notes": {
|
||||||
|
"type": "string"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Build the complete system prompt for AI filters.
|
||||||
|
///
|
||||||
|
/// Contains: role instructions, feature catalogue, few-shot examples, output rules.
|
||||||
|
/// Precomputed at startup and cached in AppState.
|
||||||
|
pub fn build_system_prompt(features: &FeaturesResponse) -> String {
|
||||||
|
let mut parts = Vec::new();
|
||||||
|
|
||||||
|
// Role and task description
|
||||||
|
parts.push(
|
||||||
|
"You are a UK property search assistant. \
|
||||||
|
The user describes their ideal property or area in natural language. \
|
||||||
|
Translate their description into filter settings using ONLY the features listed below.\n\
|
||||||
|
\n\
|
||||||
|
Rules:\n\
|
||||||
|
- ONLY set filters the user explicitly mentioned or clearly implied.\n\
|
||||||
|
- Leave out any filter the user did not mention. Empty arrays are fine.\n\
|
||||||
|
- For numeric filters, omit \"min\" to leave the lower bound open, \
|
||||||
|
omit \"max\" to leave the upper bound open.\n\
|
||||||
|
- Use EXACT feature names from the list — spelling, capitalisation, and punctuation must match.\n\
|
||||||
|
- \"cheap\" / \"affordable\" = lower price range. \"expensive\" = higher price range.\n\
|
||||||
|
- \"low crime\" / \"safe\" = low values on crime features. \
|
||||||
|
\"quiet\" = low Noise (dB). \"green\" / \"near parks\" = high Number of parks within 2km.\n\
|
||||||
|
- When the user says a number like \"under 400k\", interpret it as 400000.\n\
|
||||||
|
- When the user says \"3 bed\" or \"3 bedroom\", use Number of bedrooms & living rooms \
|
||||||
|
(note: this counts bedrooms + living rooms combined, so 3 bed ~ min 4).\n\
|
||||||
|
- If the user mentions something that has no matching filter, put it in \"notes\" \
|
||||||
|
as a short phrase (e.g. \"No filter for: garden, sea view\"). \
|
||||||
|
If everything was matched, set \"notes\" to an empty string."
|
||||||
|
.to_string(),
|
||||||
|
);
|
||||||
|
|
||||||
|
// Feature catalogue
|
||||||
|
parts.push("\n--- AVAILABLE FEATURES ---\n".to_string());
|
||||||
|
for group in &features.groups {
|
||||||
|
parts.push(format!("## {}", group.name));
|
||||||
|
for feature in &group.features {
|
||||||
|
match feature {
|
||||||
|
FeatureInfo::Numeric {
|
||||||
|
name,
|
||||||
|
min,
|
||||||
|
max,
|
||||||
|
description,
|
||||||
|
prefix,
|
||||||
|
suffix,
|
||||||
|
..
|
||||||
|
} => {
|
||||||
|
parts.push(format!(
|
||||||
|
"- \"{}\" (numeric, {}{:.0}{} to {}{:.0}{}): {}",
|
||||||
|
name, prefix, min, suffix, prefix, max, suffix, description
|
||||||
|
));
|
||||||
|
}
|
||||||
|
FeatureInfo::Enum {
|
||||||
|
name,
|
||||||
|
values,
|
||||||
|
description,
|
||||||
|
..
|
||||||
|
} => {
|
||||||
|
parts.push(format!(
|
||||||
|
"- \"{}\" (enum, values: [{}]): {}",
|
||||||
|
name,
|
||||||
|
values
|
||||||
|
.iter()
|
||||||
|
.map(|val| format!("\"{}\"", val))
|
||||||
|
.collect::<Vec<_>>()
|
||||||
|
.join(", "),
|
||||||
|
description
|
||||||
|
));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Few-shot examples
|
||||||
|
parts.push("\n--- EXAMPLES ---\n".to_string());
|
||||||
|
|
||||||
|
parts.push(
|
||||||
|
"User: \"cheap freehold house under 400k\"\n\
|
||||||
|
Output: {\"numeric_filters\": [{\"name\": \"Last known price\", \"max\": 400000}], \
|
||||||
|
\"enum_filters\": [{\"name\": \"Leashold/Freehold\", \"values\": [\"Freehold\"]}, \
|
||||||
|
{\"name\": \"Property type\", \"values\": [\"Detached\", \"Semi-Detached\", \"Terraced\"]}], \
|
||||||
|
\"notes\": \"\"}"
|
||||||
|
.to_string(),
|
||||||
|
);
|
||||||
|
|
||||||
|
parts.push(
|
||||||
|
"\nUser: \"safe quiet area with good schools and parks\"\n\
|
||||||
|
Output: {\"numeric_filters\": [\
|
||||||
|
{\"name\": \"Violence and sexual offences (avg/yr)\", \"max\": 20}, \
|
||||||
|
{\"name\": \"Burglary (avg/yr)\", \"max\": 10}, \
|
||||||
|
{\"name\": \"Noise (dB)\", \"max\": 55}, \
|
||||||
|
{\"name\": \"Good+ primary schools within 5km\", \"min\": 5}, \
|
||||||
|
{\"name\": \"Good+ secondary schools within 5km\", \"min\": 2}, \
|
||||||
|
{\"name\": \"Number of parks within 2km\", \"min\": 3}], \
|
||||||
|
\"enum_filters\": [], \"notes\": \"\"}"
|
||||||
|
.to_string(),
|
||||||
|
);
|
||||||
|
|
||||||
|
parts.push(
|
||||||
|
"\nUser: \"3 bed flat under 300k with fast broadband near the beach\"\n\
|
||||||
|
Output: {\"numeric_filters\": [\
|
||||||
|
{\"name\": \"Last known price\", \"max\": 300000}, \
|
||||||
|
{\"name\": \"Number of bedrooms & living rooms\", \"min\": 4}, \
|
||||||
|
{\"name\": \"Max available download speed (Mbps)\", \"min\": 100}], \
|
||||||
|
\"enum_filters\": [{\"name\": \"Property type\", \"values\": [\"Flat\"]}], \
|
||||||
|
\"notes\": \"No filter for: beach proximity\"}"
|
||||||
|
.to_string(),
|
||||||
|
);
|
||||||
|
|
||||||
|
parts.push(
|
||||||
|
"\nUser: \"large family home with a garden near restaurants\"\n\
|
||||||
|
Output: {\"numeric_filters\": [\
|
||||||
|
{\"name\": \"Total floor area (sqm)\", \"min\": 100}, \
|
||||||
|
{\"name\": \"Number of bedrooms & living rooms\", \"min\": 5}, \
|
||||||
|
{\"name\": \"Number of restaurants within 2km\", \"min\": 10}], \
|
||||||
|
\"enum_filters\": [{\"name\": \"Property type\", \
|
||||||
|
\"values\": [\"Detached\", \"Semi-Detached\"]}], \
|
||||||
|
\"notes\": \"No filter for: garden\"}"
|
||||||
|
.to_string(),
|
||||||
|
);
|
||||||
|
|
||||||
|
// Output format reminder
|
||||||
|
parts.push(
|
||||||
|
"\n--- OUTPUT FORMAT ---\n\
|
||||||
|
{\"numeric_filters\": [...], \"enum_filters\": [...], \"notes\": \"...\"}\n\
|
||||||
|
Respond with ONLY the JSON object. No explanation."
|
||||||
|
.to_string(),
|
||||||
|
);
|
||||||
|
|
||||||
|
parts.join("\n")
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn post_ai_filters(
|
||||||
|
state: Arc<AppState>,
|
||||||
|
Json(req): Json<AiFiltersRequest>,
|
||||||
|
) -> Result<Json<AiFiltersResponse>, (StatusCode, String)> {
|
||||||
|
info!(query = %req.query, "POST /api/ai-filters");
|
||||||
|
|
||||||
|
// Use Ollama native API with structured output
|
||||||
|
let url = format!("{}/api/chat", state.ollama_url);
|
||||||
|
let body = json!({
|
||||||
|
"model": state.ollama_model,
|
||||||
|
"messages": [
|
||||||
|
{ "role": "system", "content": state.ai_filters_system_prompt },
|
||||||
|
{ "role": "user", "content": req.query }
|
||||||
|
],
|
||||||
|
"stream": false,
|
||||||
|
"format": state.ai_filters_schema,
|
||||||
|
"options": {
|
||||||
|
"temperature": AI_FILTERS_TEMPERATURE,
|
||||||
|
"num_predict": AI_FILTERS_MAX_TOKENS,
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
let json_resp = ollama_chat(&state.http_client, &url, &body).await?;
|
||||||
|
let content = extract_ollama_content(&json_resp)?;
|
||||||
|
|
||||||
|
let content = strip_think_blocks(content);
|
||||||
|
let content = content.trim();
|
||||||
|
|
||||||
|
let raw: Value = serde_json::from_str(content).map_err(|err| {
|
||||||
|
warn!(error = %err, content = %content, "Failed to parse LLM JSON output");
|
||||||
|
(
|
||||||
|
StatusCode::BAD_GATEWAY,
|
||||||
|
format!("Failed to parse LLM output as JSON: {}", err),
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
|
||||||
|
// Validate and convert to FeatureFilters format
|
||||||
|
let filters = validate_and_convert(&raw, &state.features_response);
|
||||||
|
let notes = raw
|
||||||
|
.get("notes")
|
||||||
|
.and_then(|val| val.as_str())
|
||||||
|
.unwrap_or("")
|
||||||
|
.to_string();
|
||||||
|
|
||||||
|
Ok(Json(AiFiltersResponse { filters, notes }))
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Validate LLM output against feature metadata and convert to FeatureFilters format.
|
||||||
|
///
|
||||||
|
/// Input format (array-based, grammar-friendly):
|
||||||
|
/// ```json
|
||||||
|
/// {
|
||||||
|
/// "numeric_filters": [{"name": "Last known price", "min": 0, "max": 300000}],
|
||||||
|
/// "enum_filters": [{"name": "Leashold/Freehold", "values": ["Freehold"]}]
|
||||||
|
/// }
|
||||||
|
/// ```
|
||||||
|
///
|
||||||
|
/// Output format (FeatureFilters):
|
||||||
|
/// ```json
|
||||||
|
/// { "Last known price": [0, 300000], "Leashold/Freehold": ["Freehold"] }
|
||||||
|
/// ```
|
||||||
|
fn validate_and_convert(raw: &Value, features: &FeaturesResponse) -> Value {
|
||||||
|
let mut result = serde_json::Map::new();
|
||||||
|
|
||||||
|
// Build lookup maps from feature metadata
|
||||||
|
let mut numeric_features: rustc_hash::FxHashMap<&str, (f32, f32)> =
|
||||||
|
rustc_hash::FxHashMap::default();
|
||||||
|
let mut enum_features: rustc_hash::FxHashMap<&str, &[String]> =
|
||||||
|
rustc_hash::FxHashMap::default();
|
||||||
|
|
||||||
|
for group in &features.groups {
|
||||||
|
for feature in &group.features {
|
||||||
|
match feature {
|
||||||
|
FeatureInfo::Numeric { name, min, max, .. } => {
|
||||||
|
numeric_features.insert(name, (*min, *max));
|
||||||
|
}
|
||||||
|
FeatureInfo::Enum { name, values, .. } => {
|
||||||
|
enum_features.insert(name, values);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Process numeric filters
|
||||||
|
if let Some(arr) = raw.get("numeric_filters").and_then(|val| val.as_array()) {
|
||||||
|
for item in arr {
|
||||||
|
let name = match item.get("name").and_then(|val| val.as_str()) {
|
||||||
|
Some(name) => name,
|
||||||
|
None => continue,
|
||||||
|
};
|
||||||
|
let (feat_min, feat_max) = match numeric_features.get(name) {
|
||||||
|
Some(range) => *range,
|
||||||
|
None => continue,
|
||||||
|
};
|
||||||
|
let filter_min = item
|
||||||
|
.get("min")
|
||||||
|
.and_then(|val| val.as_f64())
|
||||||
|
.map(|num| num.max(feat_min as f64).min(feat_max as f64) as f32)
|
||||||
|
.unwrap_or(feat_min);
|
||||||
|
let filter_max = item
|
||||||
|
.get("max")
|
||||||
|
.and_then(|val| val.as_f64())
|
||||||
|
.map(|num| num.max(feat_min as f64).min(feat_max as f64) as f32)
|
||||||
|
.unwrap_or(feat_max);
|
||||||
|
// Only include if range is narrower than full range
|
||||||
|
if filter_min > feat_min || filter_max < feat_max {
|
||||||
|
result.insert(name.to_string(), json!([filter_min, filter_max]));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Process enum filters
|
||||||
|
if let Some(arr) = raw.get("enum_filters").and_then(|val| val.as_array()) {
|
||||||
|
for item in arr {
|
||||||
|
let name = match item.get("name").and_then(|val| val.as_str()) {
|
||||||
|
Some(name) => name,
|
||||||
|
None => continue,
|
||||||
|
};
|
||||||
|
let valid_values = match enum_features.get(name) {
|
||||||
|
Some(values) => *values,
|
||||||
|
None => continue,
|
||||||
|
};
|
||||||
|
if let Some(selected) = item.get("values").and_then(|val| val.as_array()) {
|
||||||
|
let valid: Vec<&str> = selected
|
||||||
|
.iter()
|
||||||
|
.filter_map(|item| item.as_str())
|
||||||
|
.filter(|str_val| valid_values.iter().any(|known| known == str_val))
|
||||||
|
.collect();
|
||||||
|
if !valid.is_empty() && valid.len() < valid_values.len() {
|
||||||
|
result.insert(name.to_string(), json!(valid));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
Value::Object(result)
|
||||||
|
}
|
||||||
|
|
@ -3,12 +3,13 @@ use std::sync::Arc;
|
||||||
use axum::http::StatusCode;
|
use axum::http::StatusCode;
|
||||||
use axum::response::Json;
|
use axum::response::Json;
|
||||||
use serde::{Deserialize, Serialize};
|
use serde::{Deserialize, Serialize};
|
||||||
use tracing::{info, warn};
|
use tracing::info;
|
||||||
|
|
||||||
use crate::consts::{
|
use crate::consts::{
|
||||||
AREA_SUMMARY_MAX_TOKENS, AREA_SUMMARY_SYSTEM_PROMPT, AREA_SUMMARY_TEMPERATURE,
|
AREA_SUMMARY_MAX_TOKENS, AREA_SUMMARY_SYSTEM_PROMPT, AREA_SUMMARY_TEMPERATURE,
|
||||||
};
|
};
|
||||||
use crate::state::AppState;
|
use crate::state::AppState;
|
||||||
|
use crate::utils::{extract_openai_content, ollama_chat, strip_think_blocks};
|
||||||
|
|
||||||
#[derive(Deserialize)]
|
#[derive(Deserialize)]
|
||||||
pub struct NumericStat {
|
pub struct NumericStat {
|
||||||
|
|
@ -89,22 +90,6 @@ fn build_prompt(req: &AreaSummaryRequest) -> String {
|
||||||
result
|
result
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Strip `<think>...</think>` blocks from model output
|
|
||||||
pub(crate) fn strip_think_blocks(text: &str) -> String {
|
|
||||||
let mut result = String::new();
|
|
||||||
let mut remaining = text;
|
|
||||||
while let Some(start) = remaining.find("<think>") {
|
|
||||||
result.push_str(&remaining[..start]);
|
|
||||||
if let Some(end) = remaining[start..].find("</think>") {
|
|
||||||
remaining = &remaining[start + end + 8..];
|
|
||||||
} else {
|
|
||||||
return result;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
result.push_str(remaining);
|
|
||||||
result
|
|
||||||
}
|
|
||||||
|
|
||||||
pub async fn post_area_summary(
|
pub async fn post_area_summary(
|
||||||
state: Arc<AppState>,
|
state: Arc<AppState>,
|
||||||
Json(req): Json<AreaSummaryRequest>,
|
Json(req): Json<AreaSummaryRequest>,
|
||||||
|
|
@ -124,45 +109,8 @@ pub async fn post_area_summary(
|
||||||
"max_tokens": AREA_SUMMARY_MAX_TOKENS,
|
"max_tokens": AREA_SUMMARY_MAX_TOKENS,
|
||||||
});
|
});
|
||||||
|
|
||||||
let response = state
|
let json = ollama_chat(&state.http_client, &url, &body).await?;
|
||||||
.http_client
|
let content = extract_openai_content(&json)?;
|
||||||
.post(&url)
|
|
||||||
.json(&body)
|
|
||||||
.send()
|
|
||||||
.await
|
|
||||||
.map_err(|err| {
|
|
||||||
warn!(error = %err, "Failed to connect to Ollama");
|
|
||||||
(
|
|
||||||
StatusCode::BAD_GATEWAY,
|
|
||||||
format!("Failed to connect to Ollama: {}", err),
|
|
||||||
)
|
|
||||||
})?;
|
|
||||||
|
|
||||||
if !response.status().is_success() {
|
|
||||||
let status = response.status();
|
|
||||||
let body_text = response.text().await.unwrap_or_default();
|
|
||||||
warn!(status = %status, body = %body_text, "Ollama returned error");
|
|
||||||
return Err((
|
|
||||||
StatusCode::BAD_GATEWAY,
|
|
||||||
format!("Ollama error {}: {}", status, body_text),
|
|
||||||
));
|
|
||||||
}
|
|
||||||
|
|
||||||
let json: serde_json::Value = response.json().await.map_err(|err| {
|
|
||||||
warn!(error = %err, "Failed to parse Ollama response");
|
|
||||||
(
|
|
||||||
StatusCode::BAD_GATEWAY,
|
|
||||||
format!("Failed to parse Ollama response: {}", err),
|
|
||||||
)
|
|
||||||
})?;
|
|
||||||
|
|
||||||
let content = json
|
|
||||||
.get("choices")
|
|
||||||
.and_then(|ch| ch.get(0))
|
|
||||||
.and_then(|ch| ch.get("message"))
|
|
||||||
.and_then(|msg| msg.get("content"))
|
|
||||||
.and_then(|ct| ct.as_str())
|
|
||||||
.unwrap_or("");
|
|
||||||
|
|
||||||
let summary = strip_think_blocks(content).trim().to_string();
|
let summary = strip_think_blocks(content).trim().to_string();
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -530,13 +530,19 @@ pub async fn get_export(
|
||||||
}
|
}
|
||||||
|
|
||||||
// Column widths
|
// Column widths
|
||||||
sheet.set_column_width(0, 12).ok();
|
sheet
|
||||||
sheet.set_column_width(1, 12).ok();
|
.set_column_width(0, 12)
|
||||||
|
.map_err(|e| format!("Failed to set column width: {e}"))?;
|
||||||
|
sheet
|
||||||
|
.set_column_width(1, 12)
|
||||||
|
.map_err(|e| format!("Failed to set column width: {e}"))?;
|
||||||
for col_offset in 0..feat_indices.len() {
|
for col_offset in 0..feat_indices.len() {
|
||||||
let col = (col_offset + 2) as u16;
|
let col = (col_offset + 2) as u16;
|
||||||
let feat_name = &feature_names[feat_indices[col_offset]];
|
let feat_name = &feature_names[feat_indices[col_offset]];
|
||||||
let width = (feat_name.len() as f64 * 1.1).clamp(10.0, 30.0);
|
let width = (feat_name.len() as f64 * 1.1).clamp(10.0, 30.0);
|
||||||
sheet.set_column_width(col, width).ok();
|
sheet
|
||||||
|
.set_column_width(col, width)
|
||||||
|
.map_err(|e| format!("Failed to set column width: {e}"))?;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -35,6 +35,8 @@ pub enum FeatureInfo {
|
||||||
suffix: &'static str,
|
suffix: &'static str,
|
||||||
#[serde(skip_serializing_if = "is_false")]
|
#[serde(skip_serializing_if = "is_false")]
|
||||||
raw: bool,
|
raw: bool,
|
||||||
|
#[serde(skip_serializing_if = "is_false")]
|
||||||
|
absolute: bool,
|
||||||
},
|
},
|
||||||
#[serde(rename = "enum")]
|
#[serde(rename = "enum")]
|
||||||
Enum {
|
Enum {
|
||||||
|
|
@ -99,6 +101,7 @@ pub fn build_features_response(data: &PropertyData) -> FeaturesResponse {
|
||||||
prefix: feature_config.prefix,
|
prefix: feature_config.prefix,
|
||||||
suffix: feature_config.suffix,
|
suffix: feature_config.suffix,
|
||||||
raw: feature_config.raw,
|
raw: feature_config.raw,
|
||||||
|
absolute: feature_config.absolute,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -6,7 +6,7 @@ use axum::response::Json;
|
||||||
use rustc_hash::FxHashMap;
|
use rustc_hash::FxHashMap;
|
||||||
use serde::{Deserialize, Serialize};
|
use serde::{Deserialize, Serialize};
|
||||||
use serde_json::{Map, Value};
|
use serde_json::{Map, Value};
|
||||||
use tracing::{info, warn};
|
use tracing::info;
|
||||||
|
|
||||||
use crate::aggregation::Aggregator;
|
use crate::aggregation::Aggregator;
|
||||||
use crate::consts::MAX_CELLS_PER_REQUEST;
|
use crate::consts::MAX_CELLS_PER_REQUEST;
|
||||||
|
|
@ -33,10 +33,55 @@ pub struct HexagonParams {
|
||||||
/// When present (even if empty), only listed features are aggregated and written.
|
/// When present (even if empty), only listed features are aggregated and written.
|
||||||
/// When absent, all features are included (backward compatible).
|
/// When absent, all features are included (backward compatible).
|
||||||
fields: Option<String>,
|
fields: Option<String>,
|
||||||
/// Destination point as "lat,lon" for real-time travel time calculation via R5.
|
/// Pipe-separated travel time entries: `lat,lon,mode|lat,lon,mode`
|
||||||
destination: Option<String>,
|
/// Each entry requests travel time from hex centroids to that destination via the given mode.
|
||||||
/// Transport mode for travel time: "transit" (default), "car", or "bicycle".
|
travel: Option<String>,
|
||||||
mode: Option<String>,
|
}
|
||||||
|
|
||||||
|
struct TravelEntry {
|
||||||
|
lat: f64,
|
||||||
|
lon: f64,
|
||||||
|
mode: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
const VALID_MODES: &[&str] = &["car", "bicycle", "walking", "transit"];
|
||||||
|
|
||||||
|
/// Parse `travel` param into a list of travel entries.
|
||||||
|
/// Format: `lat,lon,mode|lat,lon,mode`
|
||||||
|
fn parse_travel_entries(s: &str) -> Result<Vec<TravelEntry>, String> {
|
||||||
|
let mut entries = Vec::new();
|
||||||
|
let mut seen_modes = Vec::new();
|
||||||
|
for segment in s.split('|') {
|
||||||
|
let parts: Vec<&str> = segment.split(',').collect();
|
||||||
|
if parts.len() != 3 {
|
||||||
|
return Err(format!(
|
||||||
|
"each travel entry must be 'lat,lon,mode', got '{}'",
|
||||||
|
segment
|
||||||
|
));
|
||||||
|
}
|
||||||
|
let lat: f64 = parts[0]
|
||||||
|
.trim()
|
||||||
|
.parse()
|
||||||
|
.map_err(|_| format!("invalid travel latitude in '{}'", segment))?;
|
||||||
|
let lon: f64 = parts[1]
|
||||||
|
.trim()
|
||||||
|
.parse()
|
||||||
|
.map_err(|_| format!("invalid travel longitude in '{}'", segment))?;
|
||||||
|
let mode = parts[2].trim().to_string();
|
||||||
|
if !VALID_MODES.contains(&mode.as_str()) {
|
||||||
|
return Err(format!(
|
||||||
|
"invalid travel mode '{}', must be one of: {}",
|
||||||
|
mode,
|
||||||
|
VALID_MODES.join(", ")
|
||||||
|
));
|
||||||
|
}
|
||||||
|
if seen_modes.contains(&mode) {
|
||||||
|
return Err(format!("duplicate travel mode '{}'", mode));
|
||||||
|
}
|
||||||
|
seen_modes.push(mode.clone());
|
||||||
|
entries.push(TravelEntry { lat, lon, mode });
|
||||||
|
}
|
||||||
|
Ok(entries)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Build feature maps from aggregated cell data, filtering to only cells that intersect the query bounds.
|
/// Build feature maps from aggregated cell data, filtering to only cells that intersect the query bounds.
|
||||||
|
|
@ -104,23 +149,6 @@ fn build_feature_maps(
|
||||||
features
|
features
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Parse "lat,lon" string into (lat, lon) tuple.
|
|
||||||
fn parse_destination(s: &str) -> Result<[f64; 2], String> {
|
|
||||||
let parts: Vec<&str> = s.split(',').collect();
|
|
||||||
if parts.len() != 2 {
|
|
||||||
return Err("destination must be 'lat,lon'".into());
|
|
||||||
}
|
|
||||||
let lat: f64 = parts[0]
|
|
||||||
.trim()
|
|
||||||
.parse()
|
|
||||||
.map_err(|_| "invalid destination latitude")?;
|
|
||||||
let lon: f64 = parts[1]
|
|
||||||
.trim()
|
|
||||||
.parse()
|
|
||||||
.map_err(|_| "invalid destination longitude")?;
|
|
||||||
Ok([lat, lon])
|
|
||||||
}
|
|
||||||
|
|
||||||
pub async fn get_hexagons(
|
pub async fn get_hexagons(
|
||||||
state: Arc<AppState>,
|
state: Arc<AppState>,
|
||||||
Query(params): Query<HexagonParams>,
|
Query(params): Query<HexagonParams>,
|
||||||
|
|
@ -141,16 +169,17 @@ pub async fn get_hexagons(
|
||||||
|
|
||||||
let field_indices = parse_field_indices(params.fields.as_deref(), &state.feature_name_to_index);
|
let field_indices = parse_field_indices(params.fields.as_deref(), &state.feature_name_to_index);
|
||||||
|
|
||||||
// Parse destination for travel time (before moving into blocking closure)
|
// Parse travel entries
|
||||||
let destination = params
|
let travel_entries = params
|
||||||
.destination
|
.travel
|
||||||
.as_deref()
|
.as_deref()
|
||||||
.map(parse_destination)
|
.filter(|s| !s.is_empty())
|
||||||
|
.map(parse_travel_entries)
|
||||||
.transpose()
|
.transpose()
|
||||||
.map_err(|e| (StatusCode::BAD_REQUEST, e))?;
|
.map_err(|e| (StatusCode::BAD_REQUEST, e))?
|
||||||
let mode = params.mode.clone().unwrap_or_else(|| "car".into());
|
.unwrap_or_default();
|
||||||
|
|
||||||
// Capture what we need for the R5 call before moving state into spawn_blocking
|
// Capture what we need for the R5 calls before moving state into spawn_blocking
|
||||||
let r5_url = state.r5_url.clone();
|
let r5_url = state.r5_url.clone();
|
||||||
let http_client = state.http_client.clone();
|
let http_client = state.http_client.clone();
|
||||||
|
|
||||||
|
|
@ -250,14 +279,12 @@ pub async fn get_hexagons(
|
||||||
.map_err(|error| (StatusCode::INTERNAL_SERVER_ERROR, error.to_string()))?
|
.map_err(|error| (StatusCode::INTERNAL_SERVER_ERROR, error.to_string()))?
|
||||||
.map_err(|error| (StatusCode::INTERNAL_SERVER_ERROR, error))?;
|
.map_err(|error| (StatusCode::INTERNAL_SERVER_ERROR, error))?;
|
||||||
|
|
||||||
// If a destination was requested and R5 is configured, fetch travel times.
|
// If travel entries were requested and R5 is configured, fetch travel times concurrently.
|
||||||
if let Some(dest) = destination {
|
if !travel_entries.is_empty() {
|
||||||
if r5_url.is_empty() {
|
let url = r5_url.as_deref().ok_or((
|
||||||
return Err((
|
StatusCode::SERVICE_UNAVAILABLE,
|
||||||
StatusCode::SERVICE_UNAVAILABLE,
|
"Travel time queries require routing service (R5_URL not configured)".into(),
|
||||||
"Travel time queries require routing service (R5_URL not configured)".into(),
|
))?;
|
||||||
));
|
|
||||||
}
|
|
||||||
|
|
||||||
// Collect hex centroids
|
// Collect hex centroids
|
||||||
let origins: Vec<[f64; 2]> = response
|
let origins: Vec<[f64; 2]> = response
|
||||||
|
|
@ -267,39 +294,56 @@ pub async fn get_hexagons(
|
||||||
let lat = f
|
let lat = f
|
||||||
.get("lat")
|
.get("lat")
|
||||||
.and_then(|v| v.as_f64())
|
.and_then(|v| v.as_f64())
|
||||||
.unwrap_or(0.0);
|
.expect("lat must be present in feature map");
|
||||||
let lon = f
|
let lon = f
|
||||||
.get("lon")
|
.get("lon")
|
||||||
.and_then(|v| v.as_f64())
|
.and_then(|v| v.as_f64())
|
||||||
.unwrap_or(0.0);
|
.expect("lon must be present in feature map");
|
||||||
[lat, lon]
|
[lat, lon]
|
||||||
})
|
})
|
||||||
.collect();
|
.collect();
|
||||||
|
|
||||||
match fetch_travel_times(&http_client, &r5_url, origins, dest, &mode).await {
|
// Fire concurrent R5 calls for each travel entry
|
||||||
Ok(travel_times) => {
|
let mut handles = Vec::with_capacity(travel_entries.len());
|
||||||
for (feature, tt) in response.features.iter_mut().zip(travel_times) {
|
for entry in &travel_entries {
|
||||||
match tt {
|
let client = http_client.clone();
|
||||||
Some(minutes) => {
|
let url = url.to_string();
|
||||||
if let Some(num) = serde_json::Number::from_f64(minutes) {
|
let origins = origins.clone();
|
||||||
feature.insert("travel_time".into(), Value::Number(num));
|
let dest = [entry.lat, entry.lon];
|
||||||
}
|
let mode = entry.mode.clone();
|
||||||
}
|
handles.push(tokio::spawn(async move {
|
||||||
None => {
|
fetch_travel_times(&client, &url, origins, dest, &mode).await
|
||||||
feature.insert("travel_time".into(), Value::Null);
|
}));
|
||||||
|
}
|
||||||
|
|
||||||
|
let mut results = Vec::with_capacity(handles.len());
|
||||||
|
for handle in handles {
|
||||||
|
results.push(handle.await);
|
||||||
|
}
|
||||||
|
for (entry, result) in travel_entries.iter().zip(results) {
|
||||||
|
let travel_times = result
|
||||||
|
.map_err(|e| (StatusCode::INTERNAL_SERVER_ERROR, e.to_string()))?
|
||||||
|
.map_err(|err| (StatusCode::BAD_GATEWAY, err))?;
|
||||||
|
|
||||||
|
let field_name = format!("travel_time_{}", entry.mode);
|
||||||
|
for (feature, tt) in response.features.iter_mut().zip(&travel_times) {
|
||||||
|
match tt {
|
||||||
|
Some(minutes) => {
|
||||||
|
if let Some(num) = serde_json::Number::from_f64(*minutes) {
|
||||||
|
feature.insert(field_name.clone(), Value::Number(num));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
None => {
|
||||||
|
feature.insert(field_name.clone(), Value::Null);
|
||||||
|
}
|
||||||
}
|
}
|
||||||
info!(
|
|
||||||
hexagons = response.features.len(),
|
|
||||||
destination = format_args!("{},{}", dest[0], dest[1]),
|
|
||||||
mode = mode,
|
|
||||||
"Travel times merged"
|
|
||||||
);
|
|
||||||
}
|
|
||||||
Err(err) => {
|
|
||||||
warn!("Travel time query failed, returning hexagons without travel_time: {}", err);
|
|
||||||
}
|
}
|
||||||
|
info!(
|
||||||
|
hexagons = response.features.len(),
|
||||||
|
destination = format_args!("{},{}", entry.lat, entry.lon),
|
||||||
|
mode = entry.mode,
|
||||||
|
"Travel times merged"
|
||||||
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -53,11 +53,14 @@ pub async fn proxy_to_pocketbase(state: Arc<AppState>, req: Request) -> impl Int
|
||||||
if name == "transfer-encoding" {
|
if name == "transfer-encoding" {
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
response = response.header(
|
match HeaderName::from_bytes(name.as_ref()) {
|
||||||
HeaderName::from_bytes(name.as_ref())
|
Ok(header_name) => {
|
||||||
.unwrap_or(HeaderName::from_static("x-invalid")),
|
response = response.header(header_name, value.clone());
|
||||||
value.clone(),
|
}
|
||||||
);
|
Err(err) => {
|
||||||
|
warn!(header = ?name, error = %err, "Skipping unparseable upstream header");
|
||||||
|
}
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
match upstream.bytes().await {
|
match upstream.bytes().await {
|
||||||
|
|
|
||||||
|
|
@ -14,6 +14,8 @@ pub struct PlaceResult {
|
||||||
place_type: String,
|
place_type: String,
|
||||||
lat: f32,
|
lat: f32,
|
||||||
lon: f32,
|
lon: f32,
|
||||||
|
#[serde(skip_serializing_if = "Option::is_none")]
|
||||||
|
city: Option<String>,
|
||||||
}
|
}
|
||||||
|
|
||||||
#[derive(Serialize)]
|
#[derive(Serialize)]
|
||||||
|
|
@ -24,7 +26,7 @@ pub struct PlacesResponse {
|
||||||
#[derive(Deserialize)]
|
#[derive(Deserialize)]
|
||||||
#[allow(clippy::min_ident_chars)]
|
#[allow(clippy::min_ident_chars)]
|
||||||
pub struct PlacesParams {
|
pub struct PlacesParams {
|
||||||
q: Option<String>,
|
q: String,
|
||||||
limit: Option<usize>,
|
limit: Option<usize>,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -32,10 +34,11 @@ pub async fn get_places(
|
||||||
state: Arc<AppState>,
|
state: Arc<AppState>,
|
||||||
Query(params): Query<PlacesParams>,
|
Query(params): Query<PlacesParams>,
|
||||||
) -> Result<Json<PlacesResponse>, (StatusCode, String)> {
|
) -> Result<Json<PlacesResponse>, (StatusCode, String)> {
|
||||||
let query = params
|
let query = if params.q.is_empty() {
|
||||||
.q
|
return Err((StatusCode::BAD_REQUEST, "'q' must not be empty".into()));
|
||||||
.filter(|val| !val.is_empty())
|
} else {
|
||||||
.ok_or((StatusCode::BAD_REQUEST, "Missing 'q' parameter".to_string()))?;
|
params.q
|
||||||
|
};
|
||||||
|
|
||||||
let limit = params.limit.unwrap_or(7).min(20);
|
let limit = params.limit.unwrap_or(7).min(20);
|
||||||
|
|
||||||
|
|
@ -45,26 +48,37 @@ pub async fn get_places(
|
||||||
let pd = &state.place_data;
|
let pd = &state.place_data;
|
||||||
|
|
||||||
// Linear scan — ~50-100k rows, <1ms
|
// Linear scan — ~50-100k rows, <1ms
|
||||||
let mut matches: Vec<(usize, bool, u8, usize)> = pd
|
// Tuple: (row_idx, is_exact, is_prefix, type_rank, population, name_len)
|
||||||
|
let mut matches: Vec<(usize, bool, bool, u8, u32, usize)> = pd
|
||||||
.name_lower
|
.name_lower
|
||||||
.iter()
|
.iter()
|
||||||
.enumerate()
|
.enumerate()
|
||||||
.filter_map(|(idx, name)| {
|
.filter_map(|(idx, name)| {
|
||||||
if name.contains(&query_lower) {
|
if name.contains(&query_lower) {
|
||||||
|
let is_exact = name.len() == query_lower.len();
|
||||||
let is_prefix = name.starts_with(&query_lower);
|
let is_prefix = name.starts_with(&query_lower);
|
||||||
Some((idx, is_prefix, pd.type_rank[idx], pd.name[idx].len()))
|
Some((
|
||||||
|
idx,
|
||||||
|
is_exact,
|
||||||
|
is_prefix,
|
||||||
|
pd.type_rank[idx],
|
||||||
|
pd.population[idx],
|
||||||
|
pd.name[idx].len(),
|
||||||
|
))
|
||||||
} else {
|
} else {
|
||||||
None
|
None
|
||||||
}
|
}
|
||||||
})
|
})
|
||||||
.collect();
|
.collect();
|
||||||
|
|
||||||
// Sort: prefix first, then by type rank (cities before hamlets), then shorter names first
|
// Sort: exact first, then prefix, then type rank asc, then population desc, then name length asc
|
||||||
matches.sort_unstable_by(|lhs, rhs| {
|
matches.sort_unstable_by(|lhs, rhs| {
|
||||||
rhs.1
|
rhs.1
|
||||||
.cmp(&lhs.1)
|
.cmp(&lhs.1)
|
||||||
.then(lhs.2.cmp(&rhs.2))
|
.then(rhs.2.cmp(&lhs.2))
|
||||||
.then(lhs.3.cmp(&rhs.3))
|
.then(lhs.3.cmp(&rhs.3))
|
||||||
|
.then(rhs.4.cmp(&lhs.4))
|
||||||
|
.then(lhs.5.cmp(&rhs.5))
|
||||||
});
|
});
|
||||||
|
|
||||||
matches.truncate(limit);
|
matches.truncate(limit);
|
||||||
|
|
@ -76,6 +90,7 @@ pub async fn get_places(
|
||||||
place_type: pd.place_type.get(idx).to_string(),
|
place_type: pd.place_type.get(idx).to_string(),
|
||||||
lat: pd.lat[idx],
|
lat: pd.lat[idx],
|
||||||
lon: pd.lon[idx],
|
lon: pd.lon[idx],
|
||||||
|
city: pd.city[idx].clone(),
|
||||||
})
|
})
|
||||||
.collect();
|
.collect();
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -146,6 +146,9 @@ pub async fn get_hexagon_properties(
|
||||||
}
|
}
|
||||||
});
|
});
|
||||||
|
|
||||||
|
// Sort so properties with addresses come first, unknown addresses last
|
||||||
|
matching_rows.sort_unstable_by_key(|&row| state.data.address(row).trim().is_empty());
|
||||||
|
|
||||||
let total = matching_rows.len();
|
let total = matching_rows.len();
|
||||||
let limit = params
|
let limit = params
|
||||||
.limit
|
.limit
|
||||||
|
|
|
||||||
84
server-rs/src/routes/streetview.rs
Normal file
84
server-rs/src/routes/streetview.rs
Normal file
|
|
@ -0,0 +1,84 @@
|
||||||
|
use std::sync::Arc;
|
||||||
|
|
||||||
|
use axum::http::StatusCode;
|
||||||
|
use axum::response::{IntoResponse, Json};
|
||||||
|
use serde::{Deserialize, Serialize};
|
||||||
|
use tracing::warn;
|
||||||
|
|
||||||
|
use crate::state::AppState;
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
pub struct StreetViewQuery {
|
||||||
|
lat: f64,
|
||||||
|
lon: f64,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Deserialize)]
|
||||||
|
struct GoogleMetadataResponse {
|
||||||
|
status: String,
|
||||||
|
#[serde(default)]
|
||||||
|
pano_id: String,
|
||||||
|
}
|
||||||
|
|
||||||
|
#[derive(Serialize)]
|
||||||
|
struct StreetViewResponse {
|
||||||
|
status: String,
|
||||||
|
#[serde(skip_serializing_if = "Option::is_none")]
|
||||||
|
pano_id: Option<String>,
|
||||||
|
}
|
||||||
|
|
||||||
|
pub async fn get_streetview(
|
||||||
|
state: Arc<AppState>,
|
||||||
|
query: axum::extract::Query<StreetViewQuery>,
|
||||||
|
) -> impl IntoResponse {
|
||||||
|
let url = format!(
|
||||||
|
"https://maps.googleapis.com/maps/api/streetview/metadata?location={},{}&radius=1000&source=outdoor&key={}",
|
||||||
|
query.lat, query.lon, state.google_maps_api_key
|
||||||
|
);
|
||||||
|
|
||||||
|
let resp = match state.http_client.get(&url).send().await {
|
||||||
|
Ok(r) => r,
|
||||||
|
Err(e) => {
|
||||||
|
warn!("Street View metadata request failed: {e}");
|
||||||
|
return (
|
||||||
|
StatusCode::BAD_GATEWAY,
|
||||||
|
Json(StreetViewResponse {
|
||||||
|
status: "ERROR".to_string(),
|
||||||
|
pano_id: None,
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
let meta: GoogleMetadataResponse = match resp.json().await {
|
||||||
|
Ok(m) => m,
|
||||||
|
Err(e) => {
|
||||||
|
warn!("Failed to parse Street View metadata: {e}");
|
||||||
|
return (
|
||||||
|
StatusCode::BAD_GATEWAY,
|
||||||
|
Json(StreetViewResponse {
|
||||||
|
status: "ERROR".to_string(),
|
||||||
|
pano_id: None,
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
if meta.status == "OK" {
|
||||||
|
(
|
||||||
|
StatusCode::OK,
|
||||||
|
Json(StreetViewResponse {
|
||||||
|
status: "OK".to_string(),
|
||||||
|
pano_id: Some(meta.pano_id),
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
} else {
|
||||||
|
(
|
||||||
|
StatusCode::OK,
|
||||||
|
Json(StreetViewResponse {
|
||||||
|
status: meta.status,
|
||||||
|
pano_id: None,
|
||||||
|
}),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
@ -35,7 +35,6 @@ pub async fn get_tile(
|
||||||
|
|
||||||
#[derive(Deserialize)]
|
#[derive(Deserialize)]
|
||||||
pub struct StyleParams {
|
pub struct StyleParams {
|
||||||
#[serde(default)]
|
|
||||||
theme: Option<String>,
|
theme: Option<String>,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -43,26 +42,26 @@ pub async fn get_style(
|
||||||
State(reader): State<Arc<TileReader>>,
|
State(reader): State<Arc<TileReader>>,
|
||||||
headers: HeaderMap,
|
headers: HeaderMap,
|
||||||
Query(params): Query<StyleParams>,
|
Query(params): Query<StyleParams>,
|
||||||
) -> Response {
|
) -> Result<Response, (StatusCode, String)> {
|
||||||
let is_dark = params.theme.as_deref() == Some("dark");
|
let is_dark = params.theme.as_deref() == Some("dark");
|
||||||
|
|
||||||
// Metadata is returned as a JSON string
|
// Metadata is returned as a JSON string
|
||||||
let metadata_str = match reader.get_metadata().await {
|
let metadata_str = reader.get_metadata().await.map_err(|err| {
|
||||||
Ok(meta) => meta,
|
warn!(error = %err, "Failed to get PMTiles metadata");
|
||||||
Err(err) => {
|
(
|
||||||
warn!(error = %err, "Failed to get PMTiles metadata");
|
StatusCode::INTERNAL_SERVER_ERROR,
|
||||||
return StatusCode::INTERNAL_SERVER_ERROR.into_response();
|
format!("Failed to get PMTiles metadata: {err}"),
|
||||||
}
|
)
|
||||||
};
|
})?;
|
||||||
|
|
||||||
// Parse the JSON string
|
// Parse the JSON string
|
||||||
let metadata: serde_json::Value = match serde_json::from_str(&metadata_str) {
|
let metadata: serde_json::Value = serde_json::from_str(&metadata_str).map_err(|err| {
|
||||||
Ok(val) => val,
|
warn!(error = %err, "Failed to parse PMTiles metadata JSON");
|
||||||
Err(err) => {
|
(
|
||||||
warn!(error = %err, "Failed to parse PMTiles metadata JSON");
|
StatusCode::INTERNAL_SERVER_ERROR,
|
||||||
serde_json::Value::Object(serde_json::Map::new())
|
format!("Failed to parse PMTiles metadata: {err}"),
|
||||||
}
|
)
|
||||||
};
|
})?;
|
||||||
|
|
||||||
// Extract tilestats for layer info if available
|
// Extract tilestats for layer info if available
|
||||||
let layers: Vec<serde_json::Value> = metadata
|
let layers: Vec<serde_json::Value> = metadata
|
||||||
|
|
@ -75,16 +74,19 @@ pub async fn get_style(
|
||||||
let host = headers
|
let host = headers
|
||||||
.get(header::HOST)
|
.get(header::HOST)
|
||||||
.and_then(|hv| hv.to_str().ok())
|
.and_then(|hv| hv.to_str().ok())
|
||||||
.unwrap_or("localhost:8001");
|
.ok_or((
|
||||||
|
StatusCode::BAD_REQUEST,
|
||||||
|
"Missing Host header".into(),
|
||||||
|
))?;
|
||||||
let tile_url = format!("http://{}/api/tiles/{{z}}/{{x}}/{{y}}", host);
|
let tile_url = format!("http://{}/api/tiles/{{z}}/{{x}}/{{y}}", host);
|
||||||
let style = build_style(is_dark, &layers, &tile_url);
|
let style = build_style(is_dark, &layers, &tile_url);
|
||||||
|
|
||||||
(
|
Ok((
|
||||||
StatusCode::OK,
|
StatusCode::OK,
|
||||||
[(header::CONTENT_TYPE, "application/json")],
|
[(header::CONTENT_TYPE, "application/json")],
|
||||||
serde_json::to_string(&style).unwrap(),
|
serde_json::to_string(&style).unwrap(),
|
||||||
)
|
)
|
||||||
.into_response()
|
.into_response())
|
||||||
}
|
}
|
||||||
|
|
||||||
fn build_style(is_dark: bool, layers: &[serde_json::Value], tile_url: &str) -> serde_json::Value {
|
fn build_style(is_dark: bool, layers: &[serde_json::Value], tile_url: &str) -> serde_json::Value {
|
||||||
|
|
|
||||||
|
|
@ -44,12 +44,14 @@ pub struct AppState {
|
||||||
pub ollama_url: String,
|
pub ollama_url: String,
|
||||||
/// Ollama model name for area summaries (e.g. gemma3:12b)
|
/// Ollama model name for area summaries (e.g. gemma3:12b)
|
||||||
pub ollama_model: String,
|
pub ollama_model: String,
|
||||||
/// R5 routing service URL for all travel times (empty = disabled)
|
/// R5 routing service URL for all travel times (None = disabled)
|
||||||
pub r5_url: String,
|
pub r5_url: Option<String>,
|
||||||
/// Token validation cache (60s TTL)
|
/// Token validation cache (60s TTL)
|
||||||
pub token_cache: Arc<TokenCache>,
|
pub token_cache: Arc<TokenCache>,
|
||||||
/// JSON schema for Ollama structured output in AI filters
|
/// JSON schema for Ollama structured output in AI filters
|
||||||
pub ai_filters_schema: serde_json::Value,
|
pub ai_filters_schema: serde_json::Value,
|
||||||
/// Feature listing portion of the AI filters prompt
|
/// Complete system prompt for AI filters (features + examples + instructions)
|
||||||
pub ai_filters_feature_prompt: String,
|
pub ai_filters_system_prompt: String,
|
||||||
|
/// Google Maps API key for Street View metadata lookups
|
||||||
|
pub google_maps_api_key: String,
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -6,4 +6,4 @@ mod llm;
|
||||||
pub use grid_index::GridIndex;
|
pub use grid_index::GridIndex;
|
||||||
pub use hash::{generate_priorities, splitmix64_hash};
|
pub use hash::{generate_priorities, splitmix64_hash};
|
||||||
pub use interned_column::InternedColumn;
|
pub use interned_column::InternedColumn;
|
||||||
pub use llm::strip_think_blocks;
|
pub use llm::{extract_ollama_content, extract_openai_content, ollama_chat, strip_think_blocks};
|
||||||
|
|
|
||||||
|
|
@ -1,3 +1,75 @@
|
||||||
|
use axum::http::StatusCode;
|
||||||
|
use serde_json::Value;
|
||||||
|
use tracing::warn;
|
||||||
|
|
||||||
|
pub type LlmError = (StatusCode, String);
|
||||||
|
|
||||||
|
/// Send a chat request to Ollama and return the parsed JSON response.
|
||||||
|
///
|
||||||
|
/// Handles connection errors, non-success status codes, and JSON parse failures
|
||||||
|
/// uniformly as `BAD_GATEWAY` errors.
|
||||||
|
pub async fn ollama_chat(
|
||||||
|
client: &reqwest::Client,
|
||||||
|
url: &str,
|
||||||
|
body: &Value,
|
||||||
|
) -> Result<Value, LlmError> {
|
||||||
|
let response = client.post(url).json(body).send().await.map_err(|err| {
|
||||||
|
warn!(error = %err, "Failed to connect to Ollama");
|
||||||
|
(
|
||||||
|
StatusCode::BAD_GATEWAY,
|
||||||
|
format!("Failed to connect to Ollama: {}", err),
|
||||||
|
)
|
||||||
|
})?;
|
||||||
|
|
||||||
|
if !response.status().is_success() {
|
||||||
|
let status = response.status();
|
||||||
|
let body_text = response.text().await.unwrap_or_default();
|
||||||
|
warn!(status = %status, body = %body_text, "Ollama returned error");
|
||||||
|
return Err((
|
||||||
|
StatusCode::BAD_GATEWAY,
|
||||||
|
format!("Ollama error {}: {}", status, body_text),
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
response.json().await.map_err(|err| {
|
||||||
|
warn!(error = %err, "Failed to parse Ollama response");
|
||||||
|
(
|
||||||
|
StatusCode::BAD_GATEWAY,
|
||||||
|
format!("Failed to parse Ollama response: {}", err),
|
||||||
|
)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Extract content from OpenAI-compatible response (`choices[0].message.content`)
|
||||||
|
pub fn extract_openai_content(json: &Value) -> Result<&str, LlmError> {
|
||||||
|
json.get("choices")
|
||||||
|
.and_then(|ch| ch.get(0))
|
||||||
|
.and_then(|ch| ch.get("message"))
|
||||||
|
.and_then(|msg| msg.get("content"))
|
||||||
|
.and_then(|ct| ct.as_str())
|
||||||
|
.ok_or_else(|| {
|
||||||
|
warn!("Malformed OpenAI response: missing choices[0].message.content");
|
||||||
|
(
|
||||||
|
StatusCode::BAD_GATEWAY,
|
||||||
|
"Malformed LLM response: missing choices[0].message.content".into(),
|
||||||
|
)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Extract content from Ollama native response (`message.content`)
|
||||||
|
pub fn extract_ollama_content(json: &Value) -> Result<&str, LlmError> {
|
||||||
|
json.get("message")
|
||||||
|
.and_then(|msg| msg.get("content"))
|
||||||
|
.and_then(|ct| ct.as_str())
|
||||||
|
.ok_or_else(|| {
|
||||||
|
warn!("Malformed Ollama response: missing message.content");
|
||||||
|
(
|
||||||
|
StatusCode::BAD_GATEWAY,
|
||||||
|
"Malformed LLM response: missing message.content".into(),
|
||||||
|
)
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
/// Strip `<think>...</think>` blocks from model output
|
/// Strip `<think>...</think>` blocks from model output
|
||||||
pub fn strip_think_blocks(text: &str) -> String {
|
pub fn strip_think_blocks(text: &str) -> String {
|
||||||
let mut result = String::new();
|
let mut result = String::new();
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue