perfect-postcode/finder/README.md

# Finder: property listing scraper

Scrapes Greater-London sale listings from **Rightmove**, **OnTheMarket**, and
**Zoopla**, recovers each property's true full postcode, and writes a single
parquet (`data/online_listings_buy.parquet`) that the rest of the app consumes
(after a separate enrich step, see [Output](#output)).

`main.py` is the only entry point; everything else is library code.

---

## How it works (and why it's careful about postcodes)

Every portal's **search** API exposes only an *outcode*-level address (e.g.
`"…, London, SW9"`) plus map coordinates, never the full unit postcode. The
full postcode lives on each listing's **detail page**, so the scraper fetches
detail pages to recover it, and only trusts a detail postcode when its outcode
agrees with the coordinate-nearest postcode (so a stale/wrong value can never
silently relocate a listing). When no trustworthy detail postcode is found, it
falls back to the coordinate-nearest postcode. See the module docstrings in
`rightmove.py`, `onthemarket.py`, and `zoopla.py` for the per-portal data model.

Detail fetching is the dominant cost, so it is:

- **cached across runs**: `data/detail_cache/{source}.json` maps listing id →
  recovered postcode; a re-run only fetches *newly-appeared* listings;
- **fetched concurrently** for the HTTP portals (Rightmove, OnTheMarket), bounded
  by a shared global rate limiter so the VPN egress stays polite;
- **gated and capped** per outcode.

See [Performance](#performance--caching).

---

## Prerequisites

The scraper egresses through a VPN. There are two supported ways to provide it:

- **Shared netns (compose, recommended):** an **external `media_gluetun`**
  container (qmcgaw/gluetun) must already be running on the host. It is managed
  by a *different* compose; `finder/docker-compose.yml` attaches to its network
  namespace via `network_mode: "container:media_gluetun"`.
- **HTTP proxy (standalone):** reach a Gluetun HTTP proxy at `GLUETUN_PROXY`
  (default `http://gluetun:8888`), or set `GLUETUN_PROXY=""` for a direct,
  un-tunnelled connection.

Also required: the ARCGIS postcode parquet at `../property-data/arcgis_data.parquet`
(override with `ARCGIS_PATH`).

---

## Running

### Docker Compose (recommended, the only way that does Zoopla)

`finder/docker-compose.yml` brings up the scraper plus **FlareSolverr** (which
solves Zoopla's Cloudflare challenge), both sharing `media_gluetun`'s netns. This
is the intended production-like path.

```bash
cd finder

# Start the sidecars (finder stays up via `sleep infinity`).
docker compose up -d --build flaresolverr finder

# Run scrapes inside the container (uv run uses the image's /opt/venv):
docker compose exec finder uv run python main.py --source all
docker compose exec finder uv run python main.py --source zoopla --outcodes SW9 --test

docker compose down
```

> If a leftover `finder_flaresolverr` container exists from earlier testing,
> remove it first: `docker rm -f finder_flaresolverr`.

In this setup `GLUETUN_PROXY=""` (the shared netns already tunnels everything),
`ZOOPLA_FETCHER=flaresolverr`, and `DATA_DIR` / `ARCGIS_PATH` are preset by the
compose file.

### Standalone (quick Rightmove / OnTheMarket dev runs)

Zoopla needs FlareSolverr, so standalone is for the HTTP portals. You just need
a venv and VPN reachability.

```bash
cd finder

# One-time: create the venv from the lockfile.
uv sync --frozen          # creates .venv with httpx, polars, fake-useragent, …

# Small, safe run into a temp dir (does NOT touch real data/):
.venv/bin/python main.py --source rightmove --outcodes SW9 \
  --max-properties-per-source 20 --output-dir /tmp/finder-smoke

# Go direct instead of via the gluetun proxy hostname:
GLUETUN_PROXY="" .venv/bin/python main.py --source onthemarket --outcodes SW9 \
  --output-dir /tmp/finder-smoke
```

(`uv run python main.py …` works too and resolves the env automatically.)

---

## CLI reference (`main.py`)

| Flag | Default | Meaning |
|------|---------|---------|
| `--source rightmove,onthemarket` | `all` | Comma-separated portal(s): any of `rightmove`, `onthemarket`, `zoopla`, or `all`. |
| `--outcodes SW9,E14,BR1` | none | Specific outcodes (must be Greater-London-ish). Otherwise the full London set is loaded from ARCGIS. |
| `--limit-outcodes N` | none | Cap the number of outcodes (quick smoke). |
| `--max-properties-per-source N` | none | Stop each source after N transformed listings. |
| `--output-dir DIR` | `data/` | Where the parquet (and `detail_cache/`) are written. |
| `--test` | off | ~10 likely-London outcodes, ≤100 listings/source, writes to `data/test/`. |

> **Always pass `--output-dir /tmp/...` for testing**: the default `data/` holds
> the real listings the app consumes.

### Stopping a run

`Ctrl+C` (SIGINT), or `docker stop` (SIGTERM), triggers a **graceful
shutdown**: every source stops at its next outcode boundary, in-flight delays
and retry backoffs wake immediately, and the run still persists the detail
caches and writes the listings collected so far before exiting (code `130`).
Press `Ctrl+C` a second time to force-quit. See `shutdown.py`.

---

## Sources & what each needs

| Source | Transport | Needs FlareSolverr? | Concurrency | Notes |
|--------|-----------|---------------------|-------------|-------|
| Rightmove | plain httpx | no | concurrent detail fetches | main path |
| OnTheMarket | plain httpx | no | concurrent detail fetches | `__NEXT_DATA__` JSON |
| Zoopla | browser / FlareSolverr | **yes** (`ZOOPLA_FETCHER=flaresolverr`, default) | serial (browser-bound) | Cloudflare-protected; skipped gracefully if FlareSolverr is unavailable |

Rightmove and OnTheMarket run **concurrently in worker threads**; Zoopla runs on
the **main thread** (its per-outcode wall-clock guard uses `SIGALRM`, which only
fires on the main thread). One source failing never kills the others.

---

## Output

Each run writes `<output-dir>/online_listings_buy.parquet`.

A **separate enrich step** (outside `finder/`) turns that into
`online_listings_buy_enriched.parquet`, which is what the Rust backend actually
loads (`--actual-listings-path …/online_listings_buy_enriched.parquet` in the
top-level `docker-compose.yml`). That enrich/scheduling pipeline is **not**
documented here. Only the raw scrape is documented.

The top-level `docker-compose.yml` (Rust `server`, `frontend`, `pocketbase`,
`screenshot`) is the **web app**; it is downstream of the scrape and is **not**
required to run the scraper.

---

## Performance & caching

| Mechanism | Where | Effect |
|-----------|-------|--------|
| **Persistent detail cache** | `data/detail_cache/{source}.json` | A listing's postcode never changes, so a re-run reuses cached results and only fetches new listings. Delete this folder to force a full re-fetch. |
| **Concurrent detail fetches** | Rightmove, OnTheMarket | Detail pages fetched in parallel instead of one-at-a-time. |
| **Global rate limiter** | `http_client.RATE_LIMITER` | Caps the *combined* request rate across all threads/portals so concurrency stays polite. |

> **Note on the "accurate-pin skip" flag (`RIGHTMOVE_SKIP_DETAILS_FOR_ACCURATE_PINS`):**
> it is currently a **no-op**. The idea was to skip the detail fetch for listings
> the search already pins precisely (`location.pinType == "ACCURATE_POINT"`), but
> Rightmove's live search API does not include `pinType` in the payload (only
> `latitude`/`longitude`), so nothing is ever skipped. It degrades safely (no
> accuracy loss) but provides no speed-up today.

---

## Configuration

Environment variables (override the defaults in `constants.py`):

| Variable | Default | Purpose |
|----------|---------|---------|
| `DATA_DIR` | `finder/data` | Output root. |
| `ARCGIS_PATH` | `../property-data/arcgis_data.parquet` | Postcode reference data. |
| `GLUETUN_PROXY` | `http://gluetun:8888` | HTTP proxy for egress; `""` = direct. |
| `GLUETUN_CONTROL_URL` | `http://gluetun:8000` | Gluetun control API. |
| `FLARESOLVERR_URL` | `http://gluetun:8191/v1` | FlareSolverr endpoint (Zoopla). |
| `ZOOPLA_FETCHER` | `flaresolverr` | `flaresolverr` or `camoufox`. |
| `ZOOPLA_OUTCODE_TIMEOUT_SECONDS` | `300` | Per-outcode wall-clock budget for Zoopla. |
| `DETAIL_FETCH_CONCURRENCY` | `8` | Parallel detail fetches (Rightmove/OTM). |
| `REQUESTS_PER_SECOND` | `10` | Global request-rate cap. Lower it if you see `429`/`403`. |
| `RIGHTMOVE_SKIP_DETAILS_FOR_ACCURATE_PINS` | `1` | Inert today (see note above). |

Non-env code constants worth knowing (`constants.py` / `onthemarket.py`):
`RIGHTMOVE_FETCH_DETAILS`, `RIGHTMOVE_MAX_DETAILS_PER_OUTCODE` (4000),
`OTM_FETCH_DETAILS`, `OTM_MAX_DETAILS_PER_OUTCODE` (400),
`ZOOPLA_FETCH_DETAILS`, `ZOOPLA_MAX_DETAILS_PER_OUTCODE` (4000).

---

## Tests

`pytest` is not a declared dependency; run it ephemerally with uv (no project
change needed):

```bash
cd finder
uv run --with pytest pytest -q
```

---

## Repo layout

| File | Responsibility |
|------|----------------|
| `main.py` | CLI entry point: parse args, build the postcode index, call `run_scrape`. |
| `scraper.py` | Orchestration: per-source runners, provider parallelism, cache load/save, merge + write. |
| `rightmove.py` / `onthemarket.py` / `zoopla.py` | Per-portal search + detail scraping and parsing. |
| `transform.py` | Raw listing → output schema; postcode trust rules. |
| `http_client.py` | Shared httpx client, retry/backoff, and the global `RATE_LIMITER`. |
| `postcode_cache.py` | Persistent (cross-run) detail-cache load/save. |
| `spatial.py` | Grid spatial index for coordinate → nearest postcode. |
| `storage.py` | Parquet writer (server-ready column names). |
| `constants.py` | Tunables and endpoints. |
| `test_*.py` | Unit tests. |