# CLAUDE.md This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. NEVER EVER RUN GIT COMMANDS!! ## Project Overview Property Map is a full-stack geospatial application for visualizing UK property data on an interactive map. It combines Land Registry price-paid data, EPC energy certificates, postcode geolocation, TFL journey times, Index of Deprivation scores, crime statistics, ethnicity data, broadband speeds, school ratings, road noise, and OpenStreetMap POIs into a single wide parquet file, then serves aggregated H3 hexagon statistics and POI data via a Rust backend. ## Commands All commands use [Task](https://taskfile.dev) runner. Python uses `uv run`. Frontend uses `npm run` from `frontend/`. ```bash # Development servers task dev:server # Rust backend on :8001 (cargo run --release) task dev:frontend # Webpack dev server on :3001 (proxies /api to :8001) # Data pipeline task prepare # Build wide.parquet from all pre-downloaded sources # Assets task download:map-assets # Download font glyphs + twemoji PNGs into frontend/public/assets/ # Quality task lint # Lint all: Python (ruff) + TypeScript (ESLint+Prettier) + Rust (clippy+fmt) task format # Auto-fix formatting for all languages task test # Python tests (fuzzy join, haversine, POI counts) task check # Full validation: lint + build + test # Building task build:frontend # TypeScript typecheck + webpack production build task build:server # cargo build --release (NOTE: dir is wrong in Taskfile, run from server-rs/) # Granular lint/format task lint:python # uv run ruff check . task lint:frontend # eslint + prettier --check task lint:rust # cargo clippy -- -D warnings && cargo fmt --check task format:python # ruff check --fix && ruff format task format:frontend # eslint --fix + prettier --write task format:rust # cargo fmt --all ``` Running individual tests: ```bash uv run pytest pipeline/utils/test_haversine.py # Single test file uv run pytest pipeline/utils/test_haversine.py -k "test_name" # Single test ``` ## Architecture ### Data Flow ``` Raw sources → [Download scripts] → data/*.parquet → [Fuzzy join EPC ↔ Price-Paid] → epc_pp.parquet → [Merge all datasets] → wide.parquet → [Rust server loads into memory + precomputes H3 + spatial grid] → [Frontend renders deck.gl H3HexagonLayer over MapLibre GL] ``` ### Data Pipeline (`pipeline/`) Python + Polars. Two phases: 1. **Download** (`pipeline/download/`) — Each script fetches one raw dataset into `data/` 2. **Transform** (`pipeline/transform/`) — Joins and derives features: - `join_epc_pp.py` — Fuzzy-joins EPC ↔ price-paid by address within postcode buckets - `merge.py` — **Main pipeline**: joins all datasets → `wide.parquet` with human-readable column names - `transform_poi.py` — Filters POIs, maps to friendly names + emoji (exhaustive category validation) - `poi_proximity.py` — Counts POIs within 2km per postcode using 0.05° spatial grid - `crime.py` — Aggregates crime CSVs into yearly averages by LSOA **Critical: column renaming in `merge.py`** — The pipeline renames columns from snake_case to human-readable names before writing `wide.parquet`. The Rust server and frontend use **only** these human-readable names — there are no fallbacks to snake_case. Key renames: - `pp_address` → `Address per Property Register` - `postcode` → `Postcode` - `latest_price` → `Last known price` - `duration` → `Leashold/Freehold` - `total_floor_area` → `Total floor area (sqm)` - `current_energy_rating` → `Current energy rating` The server requires these exact column names at startup (will error if missing). See the full rename map in `merge.py`. ### Backend (`server-rs/`) Rust + Axum. Loads parquet into memory at startup. **Structure** (uses Rust 2018 module style — `foo.rs` + `foo/` directory, not `foo/mod.rs`): - `data.rs` + `data/` — Property and POI data loading - `parsing.rs` + `parsing/` — Filter parsing and bounds parsing - `routes.rs` + `routes/` — One file per endpoint. `properties.rs` exports shared `build_property()` used by both hexagon and postcode property endpoints - `utils.rs` + `utils/` — GridIndex, hashing, interned columns - `consts.rs` — Key constants (histogram bins, H3 range, max enum cardinality, excluded columns) **API endpoints:** - `GET /api/features` — Feature metadata with histograms and 2nd/98th percentiles - `GET /api/hexagons?resolution=&bounds=&filters=&fields=` — H3 aggregates (min/max per feature per hex), AABB-filtered to bounds - `GET /api/postcodes?bounds=&filters=&fields=` — Postcode polygon aggregates, AABB-filtered to bounds - `GET /api/postcode/:postcode` — Single postcode lookup (centroid + polygon) - `GET /api/hexagon-properties?h3=&resolution=&filters=&limit=&offset=` — Paginated properties within a hexagon - `GET /api/postcode-properties?postcode=&filters=&limit=&offset=` — Paginated properties within a postcode - `GET /api/pois?bounds=&categories=` — POIs by bounds (max 5000) - `GET /api/poi-categories` — Available POI category names Serves `frontend/dist/` as static fallback in production **only** when `--dist` is explicitly provided. When `--dist` is set, the server panics at startup if `index.html` is unreadable. When omitted (dev mode), static serving and OG injection are disabled. **Data representation (unified model):** - All features (numeric and enum): row-major flat `Vec`, NaN = null - Enum features: stored as f32 indices (0.0, 1.0, 2.0...) with `enum_values: FxHashMap>` mapping feature index → string values - String fields (address, postcode): interned/packed for memory efficiency - All CLI args are required (no hidden defaults). Optional services use `Option`: `r5_url` (travel time disabled when None), `pocketbase_admin_email`/`password` (collection auto-creation skipped when None). Required config like `ollama_model` and `public_url` must be explicitly provided via env or CLI. ### Frontend (`frontend/`) React 18 + TypeScript. deck.gl `H3HexagonLayer` over MapLibre GL. TailwindCSS. No state management library — pure React hooks. **Architecture:** - `App.tsx` — Minimal router: loads features/POI categories, handles page navigation (home/dashboard/data-sources/faq) - `MapPage.tsx` — Dashboard layout: composes map + left/right panes, uses custom hooks for all logic - Custom hooks in `hooks/` encapsulate stateful logic: - `useMapData` — Hexagon/postcode fetching, bounds, loading state, color range calculation - `useFilters` — Filter state and handlers (add/remove/change/drag/pin) - `useHexagonSelection` — Selection state, area stats, properties fetching (supports both hexagons and postcodes) - `usePOIData` — POI fetching with debounce - `usePaneResize` — Reusable pane resize handlers - `useTheme` — Theme state with localStorage persistence - `useUrlSync` — URL state synchronization **Key patterns:** - URL encodes view/filters/POI categories/active tab as query params for shareable links. Only the current format is supported — no legacy parameter parsing (old `v=`, `f=`, or tab abbreviations are not handled). `tmode` is always serialized when travel time is active (no implicit default); parsing throws if `tmode` is missing when `dest` is present. - AbortControllers cancel in-flight requests on new queries (150ms debounce) - Zoom → H3 resolution defined in `consts.ts` `ZOOM_TO_RESOLUTION_THRESHOLDS`: `<7.5→5, <9.5→6, <10.5→8, <12→9, ≥12→10` - `POSTCODE_ZOOM_THRESHOLD = 15`: below 15 shows H3 hexagons, at/above 15 shows postcode polygons - Viewport bounds computed via `getBoundsFromViewState()` in `map-utils.ts` — uses Web Mercator math with **TILE_SIZE=512** (MapLibre/deck.gl convention, NOT 256) - Properties pane uses feature names from API response (human-readable), not hardcoded field names - Proxy: dev server on :3001 proxies `/api` to :8001; also handles VS Code `/proxy/PORT` patterns **Shared UI Components (`frontend/src/components/ui/`):** - `icons/` — One file per icon (CloseIcon, InfoIcon, EyeIcon, PlusIcon, ChevronIcon, FilterIcon, LightbulbIcon, DownloadIcon, MapPinIcon, CheckIcon, ClipboardIcon, SunIcon, MoonIcon, SpinnerIcon). All accept `className` prop. **Never inline SVGs** — always extract to this folder. - `IconButton.tsx` — Reusable icon button wrapper with consistent hover states. Accepts `active` prop for teal highlight. - `SearchInput.tsx` — Styled search input with dark mode support. Used in Filters, POIPane, PropertiesPane. - `PaneHeader.tsx` — Reusable pane header with title, optional subtitle, info button, and close button. - `SelectionButtons.tsx` — "All" / "None" selection buttons for checkbox lists. - `TabButton.tsx` — Tab button with active state styling. Used in right pane tabs. - `EmptyState.tsx` — Empty state display with icon, title, description. Also exports `PaneEmptyState` for centered pane messages. - `CheckboxList.tsx` — Checkbox list with toggle logic. Variants for array and Set-based selection. **Shared Components (`frontend/src/components/`):** - `FeatureInfoPopup.tsx` — Popup showing feature name, description, detail, and "View data source" link. - `FeatureIcons.tsx` — `FeatureActions` component combining eye/info/add/remove icons for feature rows. **Shared Utilities (`frontend/src/lib/`):** - `api.ts` — `apiUrl(endpoint, params?)` builds API URLs. `logNonAbortError(label, err)` and `isAbortError(err)` for error handling. - `features.ts` — `groupFeaturesByCategory(features)` groups FeatureMeta[] by their `group` field. - `format.ts` — `formatNumber(value, decimals)` for number formatting. `calculateHistogramMean(histogram)` for weighted mean calculation. - `property-fields.ts` — `getNum(property, key)` for getting a single numeric property value. Takes exactly one key — no fallback names. When adding new UI, prefer using these shared components over inline implementations to maintain consistency. **When to extract vs inline:** - Extract to `hooks/`: Stateful logic with useState/useEffect/useCallback that can be named as a cohesive unit (e.g., `useFilters`, `useMapData`). If a component has 5+ related state variables and handlers, extract them to a hook. - Extract to page component: Layout + hook composition for a major view (e.g., `MapPage` composes `useMapData` + `useFilters` + child components). Keep App.tsx focused on routing. - Extract to `ui/` component: Repeated 3+ times with same styling (buttons, inputs, icons) - Extract to `lib/`: Pure functions used across components (formatting, calculations, lookups) - Keep inline: One-off UI specific to a single component **Component size guideline:** If a component exceeds ~300 lines, look for extraction opportunities. Large components are usually doing too much — split into hooks (for logic) and child components (for UI sections). **Naming conventions:** - UI components: PascalCase, noun-based (`TabButton`, `EmptyState`) - Utilities: camelCase verb-based (`formatNumber`, `calculateHistogramMean`) ## Frontend Design Guide (STRICT — must be followed for all UI changes) The frontend uses Tailwind's `darkMode: 'class'` strategy. The `dark` class is toggled on ``. Every visible element must have both light and dark styles. **Never add a light-only color class without its `dark:` counterpart.** Run `task build:frontend` after any UI change to verify. ### Theme System - **State**: `App.tsx` owns a `theme` state (`'light' | 'dark' | 'system'`), persisted in `localStorage` under the key `theme`, default `'system'`. - **Effective theme**: When `'system'`, resolved via `window.matchMedia('(prefers-color-scheme: dark)')`. A `change` listener re-renders on OS preference flip. - **Toggle cycle**: light → dark → system → light. Three-way, not binary. - **Flash prevention**: `index.html` contains an inline `