andras/perfect-postcode
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages 2 Wiki Activity Actions

wip

Browse source
This commit is contained in:
Andras Schmelczer 2026-06-28 11:59:44 +01:00
parent f1601257c7
commit c2070693fb
68 changed files with 2305 additions and 212 deletions
Download patch file Download diff file Expand all files Collapse all files

22
finder/README.md
View file

@ -1,9 +1,9 @@
# Finder — property listing scraper
# Finder: property listing scraper
Scrapes Greater-London sale listings from **Rightmove**, **OnTheMarket**, and
**Zoopla**, recovers each property's true full postcode, and writes a single
parquet (`data/online_listings_buy.parquet`) that the rest of the app consumes
(after a separate enrich step — see [Output](#output)).
(after a separate enrich step, see [Output](#output)).
`main.py` is the only entry point; everything else is library code.
@ -12,7 +12,7 @@ parquet (`data/online_listings_buy.parquet`) that the rest of the app consumes
## How it works (and why it's careful about postcodes)
Every portal's **search** API exposes only an *outcode*-level address (e.g.
`"…, London, SW9"`) plus map coordinates — never the full unit postcode. The
`"…, London, SW9"`) plus map coordinates, never the full unit postcode. The
full postcode lives on each listing's **detail page**, so the scraper fetches
detail pages to recover it, and only trusts a detail postcode when its outcode
agrees with the coordinate-nearest postcode (so a stale/wrong value can never
@ -22,7 +22,7 @@ falls back to the coordinate-nearest postcode. See the module docstrings in
Detail fetching is the dominant cost, so it is:
- **cached across runs** — `data/detail_cache/{source}.json` maps listing id →
- **cached across runs**: `data/detail_cache/{source}.json` maps listing id →
recovered postcode; a re-run only fetches *newly-appeared* listings;
- **fetched concurrently** for the HTTP portals (Rightmove, OnTheMarket), bounded
by a shared global rate limiter so the VPN egress stays polite;
@ -51,7 +51,7 @@ Also required: the ARCGIS postcode parquet at `../property-data/arcgis_data.parq
## Running
### Docker Compose (recommended — the only way that does Zoopla)
### Docker Compose (recommended, the only way that does Zoopla)
`finder/docker-compose.yml` brings up the scraper plus **FlareSolverr** (which
solves Zoopla's Cloudflare challenge), both sharing `media_gluetun`'s netns. This
@ -106,18 +106,18 @@ GLUETUN_PROXY="" .venv/bin/python main.py --source onthemarket --outcodes SW9 \
| Flag | Default | Meaning |
|------|---------|---------|
| `--source rightmove,onthemarket` | `all` | Comma-separated portal(s): any of `rightmove`, `onthemarket`, `zoopla`, or `all`. |
| `--outcodes SW9,E14,BR1` | — | Specific outcodes (must be Greater-London-ish). Otherwise the full London set is loaded from ARCGIS. |
| `--limit-outcodes N` | — | Cap the number of outcodes (quick smoke). |
| `--max-properties-per-source N` | — | Stop each source after N transformed listings. |
| `--outcodes SW9,E14,BR1` | none | Specific outcodes (must be Greater-London-ish). Otherwise the full London set is loaded from ARCGIS. |
| `--limit-outcodes N` | none | Cap the number of outcodes (quick smoke). |
| `--max-properties-per-source N` | none | Stop each source after N transformed listings. |
| `--output-dir DIR` | `data/` | Where the parquet (and `detail_cache/`) are written. |
| `--test` | off | ~10 likely-London outcodes, ≤100 listings/source, writes to `data/test/`. |
> **Always pass `--output-dir /tmp/...` for testing** — the default `data/` holds
> **Always pass `--output-dir /tmp/...` for testing**: the default `data/` holds
> the real listings the app consumes.
### Stopping a run
`Ctrl+C` (SIGINT) — or `docker stop` (SIGTERM) — triggers a **graceful
`Ctrl+C` (SIGINT), or `docker stop` (SIGTERM), triggers a **graceful
shutdown**: every source stops at its next outcode boundary, in-flight delays
and retry backoffs wake immediately, and the run still persists the detail
caches and writes the listings collected so far before exiting (code `130`).
@ -147,7 +147,7 @@ A **separate enrich step** (outside `finder/`) turns that into
`online_listings_buy_enriched.parquet`, which is what the Rust backend actually
loads (`--actual-listings-path …/online_listings_buy_enriched.parquet` in the
top-level `docker-compose.yml`). That enrich/scheduling pipeline is **not**
documented here — only the raw scrape is.
documented here. Only the raw scrape is documented.
The top-level `docker-compose.yml` (Rust `server`, `frontend`, `pocketbase`,
`screenshot`) is the **web app**; it is downstream of the scrape and is **not**