perfect-postcode/finder/README.md

Finder — property listing scraper

Scrapes Greater-London sale listings from Rightmove, OnTheMarket, and Zoopla, recovers each property's true full postcode, and writes a single parquet (data/online_listings_buy.parquet) that the rest of the app consumes (after a separate enrich step — see Output).

main.py is the only entry point; everything else is library code.

---

How it works (and why it's careful about postcodes)

Every portal's search API exposes only an outcode-level address (e.g. "…, London, SW9") plus map coordinates — never the full unit postcode. The full postcode lives on each listing's detail page, so the scraper fetches detail pages to recover it, and only trusts a detail postcode when its outcode agrees with the coordinate-nearest postcode (so a stale/wrong value can never silently relocate a listing). When no trustworthy detail postcode is found, it falls back to the coordinate-nearest postcode. See the module docstrings in rightmove.py, onthemarket.py, and zoopla.py for the per-portal data model.

Detail fetching is the dominant cost, so it is:

• cached across runs — data/detail_cache/{source}.json maps listing id → recovered postcode; a re-run only fetches newly-appeared listings;
• fetched concurrently for the HTTP portals (Rightmove, OnTheMarket), bounded by a shared global rate limiter so the VPN egress stays polite;
• gated and capped per outcode.

See Performance.

---

Prerequisites

The scraper egresses through a VPN. There are two supported ways to provide it:

• Shared netns (compose, recommended): an external media_gluetun container (qmcgaw/gluetun) must already be running on the host. It is managed by a different compose; finder/docker-compose.yml attaches to its network namespace via network_mode: "container:media_gluetun".
• HTTP proxy (standalone): reach a Gluetun HTTP proxy at GLUETUN_PROXY (default http://gluetun:8888), or set GLUETUN_PROXY="" for a direct, un-tunnelled connection.

Also required: the ARCGIS postcode parquet at ../property-data/arcgis_data.parquet (override with ARCGIS_PATH).

---

Running

Docker Compose (recommended — the only way that does Zoopla)

finder/docker-compose.yml brings up the scraper plus FlareSolverr (which solves Zoopla's Cloudflare challenge), both sharing media_gluetun's netns. This is the intended production-like path.

cd finder

# Start the sidecars (finder stays up via `sleep infinity`).
docker compose up -d --build flaresolverr finder

# Run scrapes inside the container (uv run uses the image's /opt/venv):
docker compose exec finder uv run python main.py --source all
docker compose exec finder uv run python main.py --source zoopla --outcodes SW9 --test

docker compose down

If a leftover finder_flaresolverr container exists from earlier testing, remove it first: docker rm -f finder_flaresolverr.

In this setup GLUETUN_PROXY="" (the shared netns already tunnels everything), ZOOPLA_FETCHER=flaresolverr, and DATA_DIR / ARCGIS_PATH are preset by the compose file.

Standalone (quick Rightmove / OnTheMarket dev runs)

Zoopla needs FlareSolverr, so standalone is for the HTTP portals. You just need a venv and VPN reachability.

cd finder

# One-time: create the venv from the lockfile.
uv sync --frozen          # creates .venv with httpx, polars, fake-useragent, …

# Small, safe run into a temp dir (does NOT touch real data/):
.venv/bin/python main.py --source rightmove --outcodes SW9 \
  --max-properties-per-source 20 --output-dir /tmp/finder-smoke

# Go direct instead of via the gluetun proxy hostname:
GLUETUN_PROXY="" .venv/bin/python main.py --source onthemarket --outcodes SW9 \
  --output-dir /tmp/finder-smoke

(uv run python main.py … works too and resolves the env automatically.)

---

CLI reference (main.py)

Flag	Default	Meaning
--source rightmove,onthemarket	all	Comma-separated portal(s): any of rightmove, onthemarket, zoopla, or all.
--outcodes SW9,E14,BR1	—	Specific outcodes (must be Greater-London-ish). Otherwise the full London set is loaded from ARCGIS.
--limit-outcodes N	—	Cap the number of outcodes (quick smoke).
--max-properties-per-source N	—	Stop each source after N transformed listings.
--output-dir DIR	data/	Where the parquet (and detail_cache/) are written.
--test	off	~10 likely-London outcodes, ≤100 listings/source, writes to data/test/.

Always pass --output-dir /tmp/... for testing — the default data/ holds the real listings the app consumes.

Stopping a run

Ctrl+C (SIGINT) — or docker stop (SIGTERM) — triggers a graceful shutdown: every source stops at its next outcode boundary, in-flight delays and retry backoffs wake immediately, and the run still persists the detail caches and writes the listings collected so far before exiting (code 130). Press Ctrl+C a second time to force-quit. See shutdown.py.

---

Sources & what each needs

Source	Transport	Needs FlareSolverr?	Concurrency	Notes
Rightmove	plain httpx	no	concurrent detail fetches	main path
OnTheMarket	plain httpx	no	concurrent detail fetches	__NEXT_DATA__ JSON
Zoopla	browser / FlareSolverr	yes (ZOOPLA_FETCHER=flaresolverr, default)	serial (browser-bound)	Cloudflare-protected; skipped gracefully if FlareSolverr is unavailable

Rightmove and OnTheMarket run concurrently in worker threads; Zoopla runs on the main thread (its per-outcode wall-clock guard uses SIGALRM, which only fires on the main thread). One source failing never kills the others.

---

Output

Each run writes <output-dir>/online_listings_buy.parquet.

A separate enrich step (outside finder/) turns that into online_listings_buy_enriched.parquet, which is what the Rust backend actually loads (--actual-listings-path …/online_listings_buy_enriched.parquet in the top-level docker-compose.yml). That enrich/scheduling pipeline is not documented here — only the raw scrape is.

The top-level docker-compose.yml (Rust server, frontend, pocketbase, screenshot) is the web app; it is downstream of the scrape and is not required to run the scraper.

---

Performance & caching

Mechanism	Where	Effect
Persistent detail cache	data/detail_cache/{source}.json	A listing's postcode never changes, so a re-run reuses cached results and only fetches new listings. Delete this folder to force a full re-fetch.
Concurrent detail fetches	Rightmove, OnTheMarket	Detail pages fetched in parallel instead of one-at-a-time.
Global rate limiter	http_client.RATE_LIMITER	Caps the combined request rate across all threads/portals so concurrency stays polite.

Note on the "accurate-pin skip" flag (RIGHTMOVE_SKIP_DETAILS_FOR_ACCURATE_PINS): it is currently a no-op. The idea was to skip the detail fetch for listings the search already pins precisely (location.pinType == "ACCURATE_POINT"), but Rightmove's live search API does not include pinType in the payload (only latitude/longitude), so nothing is ever skipped. It degrades safely (no accuracy loss) but provides no speed-up today.

---

Configuration

Environment variables (override the defaults in constants.py):

Variable	Default	Purpose
DATA_DIR	finder/data	Output root.
ARCGIS_PATH	../property-data/arcgis_data.parquet	Postcode reference data.
GLUETUN_PROXY	http://gluetun:8888	HTTP proxy for egress; "" = direct.
GLUETUN_CONTROL_URL	http://gluetun:8000	Gluetun control API.
FLARESOLVERR_URL	http://gluetun:8191/v1	FlareSolverr endpoint (Zoopla).
ZOOPLA_FETCHER	flaresolverr	flaresolverr or camoufox.
ZOOPLA_OUTCODE_TIMEOUT_SECONDS	300	Per-outcode wall-clock budget for Zoopla.
DETAIL_FETCH_CONCURRENCY	8	Parallel detail fetches (Rightmove/OTM).
REQUESTS_PER_SECOND	10	Global request-rate cap. Lower it if you see 429/403.
RIGHTMOVE_SKIP_DETAILS_FOR_ACCURATE_PINS	1	Inert today (see note above).

Non-env code constants worth knowing (constants.py / onthemarket.py): RIGHTMOVE_FETCH_DETAILS, RIGHTMOVE_MAX_DETAILS_PER_OUTCODE (4000), OTM_FETCH_DETAILS, OTM_MAX_DETAILS_PER_OUTCODE (400), ZOOPLA_FETCH_DETAILS, ZOOPLA_MAX_DETAILS_PER_OUTCODE (4000).

---

Tests

pytest is not a declared dependency; run it ephemerally with uv (no project change needed):

cd finder
uv run --with pytest pytest -q

---

Repo layout

File	Responsibility
main.py	CLI entry point: parse args, build the postcode index, call run_scrape.
scraper.py	Orchestration: per-source runners, provider parallelism, cache load/save, merge + write.
rightmove.py / onthemarket.py / zoopla.py	Per-portal search + detail scraping and parsing.
transform.py	Raw listing → output schema; postcode trust rules.
http_client.py	Shared httpx client, retry/backoff, and the global RATE_LIMITER.
postcode_cache.py	Persistent (cross-run) detail-cache load/save.
spatial.py	Grid spatial index for coordinate → nearest postcode.
storage.py	Parquet writer (server-ready column names).
constants.py	Tunables and endpoints.
test_*.py	Unit tests.