perfect-postcode/finder

Finder: property listing scraper

Scrapes Greater-London sale listings from Rightmove, OnTheMarket, and Zoopla, recovers each property's true full postcode, and writes a single parquet (data/online_listings_buy.parquet) that the rest of the app consumes (after a separate enrich step, see Output).

main.py is the only entry point; everything else is library code.

---

How it works (and why it's careful about postcodes)

Every portal's search API exposes only an outcode-level address (e.g. "…, London, SW9") plus map coordinates, never the full unit postcode. The full postcode lives on each listing's detail page, so the scraper fetches detail pages to recover it, and only trusts a detail postcode when its outcode agrees with the coordinate-nearest postcode (so a stale/wrong value can never silently relocate a listing). When no trustworthy detail postcode is found, it falls back to the coordinate-nearest postcode. See the module docstrings in rightmove.py, onthemarket.py, and zoopla.py for the per-portal data model.

Detail fetching is the dominant cost, so it is:

• cached across runs: data/detail_cache/{source}.json maps listing id → recovered postcode; a re-run only fetches newly-appeared listings;
• fetched concurrently for the HTTP portals (Rightmove, OnTheMarket), bounded by a shared global rate limiter so the VPN egress stays polite;
• gated and capped per outcode.

See Performance.

---

Prerequisites

The scraper egresses through a VPN. There are two supported ways to provide it:

• Shared netns (compose, recommended): an external media_gluetun container (qmcgaw/gluetun) must already be running on the host. It is managed by a different compose; finder/docker-compose.yml attaches to its network namespace via network_mode: "container:media_gluetun".
• HTTP proxy (standalone): reach a Gluetun HTTP proxy at GLUETUN_PROXY (default http://gluetun:8888), or set GLUETUN_PROXY="" for a direct, un-tunnelled connection.

Also required: the ARCGIS postcode parquet at ../property-data/arcgis_data.parquet (override with ARCGIS_PATH).

---

Running

Docker Compose (recommended, the only way that does Zoopla)

finder/docker-compose.yml brings up the scraper plus FlareSolverr (which solves Zoopla's Cloudflare challenge), both sharing media_gluetun's netns. This is the intended production-like path.

cd finder

# Start the sidecars (finder stays up via `sleep infinity`).
docker compose up -d --build flaresolverr finder

# Run scrapes inside the container (uv run uses the image's /opt/venv):
docker compose exec finder uv run python main.py --source all
docker compose exec finder uv run python main.py --source zoopla --outcodes SW9 --test

docker compose down

If a leftover finder_flaresolverr container exists from earlier testing, remove it first: docker rm -f finder_flaresolverr.

In this setup GLUETUN_PROXY="" (the shared netns already tunnels everything), ZOOPLA_FETCHER=flaresolverr, and DATA_DIR / ARCGIS_PATH are preset by the compose file.

Standalone (quick Rightmove / OnTheMarket dev runs)

Zoopla needs FlareSolverr, so standalone is for the HTTP portals. You just need a venv and VPN reachability.

cd finder

# One-time: create the venv from the lockfile.
uv sync --frozen          # creates .venv with httpx, polars, fake-useragent, …

# Small, safe run into a temp dir (does NOT touch real data/):
.venv/bin/python main.py --source rightmove --outcodes SW9 \
  --max-properties-per-source 20 --output-dir /tmp/finder-smoke

# Go direct instead of via the gluetun proxy hostname:
GLUETUN_PROXY="" .venv/bin/python main.py --source onthemarket --outcodes SW9 \
  --output-dir /tmp/finder-smoke

(uv run python main.py … works too and resolves the env automatically.)

---

CLI reference (main.py)

Flag	Default	Meaning
--source rightmove,onthemarket	all	Comma-separated portal(s): any of rightmove, onthemarket, zoopla, or all.
--outcodes SW9,E14,BR1	none	Specific outcodes (must be Greater-London-ish). Otherwise the full London set is loaded from ARCGIS.
--limit-outcodes N	none	Cap the number of outcodes (quick smoke).
--max-properties-per-source N	none	Stop each source after N transformed listings.
--output-dir DIR	data/	Where the parquet (and detail_cache/) are written.
--test	off	~10 likely-London outcodes, ≤100 listings/source, writes to data/test/.

Always pass --output-dir /tmp/... for testing: the default data/ holds the real listings the app consumes.

Stopping a run

Ctrl+C (SIGINT), or docker stop (SIGTERM), triggers a graceful shutdown: every source stops at its next outcode boundary, in-flight delays and retry backoffs wake immediately, and the run still persists the detail caches and writes the listings collected so far before exiting (code 130). Press Ctrl+C a second time to force-quit. See shutdown.py.

---

Sources & what each needs

Source	Transport	Needs FlareSolverr?	Concurrency	Notes
Rightmove	plain httpx	no	concurrent detail fetches	main path
OnTheMarket	plain httpx	no	concurrent detail fetches	__NEXT_DATA__ JSON
Zoopla	browser / FlareSolverr	yes (ZOOPLA_FETCHER=flaresolverr, default)	serial (browser-bound)	Cloudflare-protected; skipped gracefully if FlareSolverr is unavailable

Rightmove and OnTheMarket run concurrently in worker threads; Zoopla runs on the main thread (its per-outcode wall-clock guard uses SIGALRM, which only fires on the main thread). One source failing never kills the others.

---

Output

Each run writes <output-dir>/online_listings_buy.parquet.

A separate enrich step (outside finder/) turns that into online_listings_buy_enriched.parquet, which is what the Rust backend actually loads (--actual-listings-path …/online_listings_buy_enriched.parquet in the top-level docker-compose.yml). That enrich/scheduling pipeline is not documented here. Only the raw scrape is documented.

The top-level docker-compose.yml (Rust server, frontend, pocketbase, screenshot) is the web app; it is downstream of the scrape and is not required to run the scraper.

---

Performance & caching

Mechanism	Where	Effect
Persistent detail cache	data/detail_cache/{source}.json	A listing's postcode never changes, so a re-run reuses cached results and only fetches new listings. Delete this folder to force a full re-fetch.
Concurrent detail fetches	Rightmove, OnTheMarket	Detail pages fetched in parallel instead of one-at-a-time.
Global rate limiter	http_client.RATE_LIMITER	Caps the combined request rate across all threads/portals so concurrency stays polite.

Note on the "accurate-pin skip" flag (RIGHTMOVE_SKIP_DETAILS_FOR_ACCURATE_PINS): it is currently a no-op. The idea was to skip the detail fetch for listings the search already pins precisely (location.pinType == "ACCURATE_POINT"), but Rightmove's live search API does not include pinType in the payload (only latitude/longitude), so nothing is ever skipped. It degrades safely (no accuracy loss) but provides no speed-up today.

---

Configuration

Environment variables (override the defaults in constants.py):

Variable	Default	Purpose
DATA_DIR	finder/data	Output root.
ARCGIS_PATH	../property-data/arcgis_data.parquet	Postcode reference data.
GLUETUN_PROXY	http://gluetun:8888	HTTP proxy for egress; "" = direct.
GLUETUN_CONTROL_URL	http://gluetun:8000	Gluetun control API.
FLARESOLVERR_URL	http://gluetun:8191/v1	FlareSolverr endpoint (Zoopla).
ZOOPLA_FETCHER	flaresolverr	flaresolverr or camoufox.
ZOOPLA_OUTCODE_TIMEOUT_SECONDS	300	Per-outcode wall-clock budget for Zoopla.
DETAIL_FETCH_CONCURRENCY	8	Parallel detail fetches (Rightmove/OTM).
REQUESTS_PER_SECOND	10	Global request-rate cap. Lower it if you see 429/403.
RIGHTMOVE_SKIP_DETAILS_FOR_ACCURATE_PINS	1	Inert today (see note above).

Non-env code constants worth knowing (constants.py / onthemarket.py): RIGHTMOVE_FETCH_DETAILS, RIGHTMOVE_MAX_DETAILS_PER_OUTCODE (4000), OTM_FETCH_DETAILS, OTM_MAX_DETAILS_PER_OUTCODE (400), ZOOPLA_FETCH_DETAILS, ZOOPLA_MAX_DETAILS_PER_OUTCODE (4000).

---

Tests

pytest is not a declared dependency; run it ephemerally with uv (no project change needed):

cd finder
uv run --with pytest pytest -q

---

Repo layout

File	Responsibility
main.py	CLI entry point: parse args, build the postcode index, call run_scrape.
scraper.py	Orchestration: per-source runners, provider parallelism, cache load/save, merge + write.
rightmove.py / onthemarket.py / zoopla.py	Per-portal search + detail scraping and parsing.
transform.py	Raw listing → output schema; postcode trust rules.
http_client.py	Shared httpx client, retry/backoff, and the global RATE_LIMITER.
postcode_cache.py	Persistent (cross-run) detail-cache load/save.
spatial.py	Grid spatial index for coordinate → nearest postcode.
storage.py	Parquet writer (server-ready column names).
constants.py	Tunables and endpoints.
test_*.py	Unit tests.