wip
This commit is contained in:
parent
f1601257c7
commit
c2070693fb
68 changed files with 2305 additions and 212 deletions
|
|
@ -1,9 +1,9 @@
|
|||
# Finder — property listing scraper
|
||||
# Finder: property listing scraper
|
||||
|
||||
Scrapes Greater-London sale listings from **Rightmove**, **OnTheMarket**, and
|
||||
**Zoopla**, recovers each property's true full postcode, and writes a single
|
||||
parquet (`data/online_listings_buy.parquet`) that the rest of the app consumes
|
||||
(after a separate enrich step — see [Output](#output)).
|
||||
(after a separate enrich step, see [Output](#output)).
|
||||
|
||||
`main.py` is the only entry point; everything else is library code.
|
||||
|
||||
|
|
@ -12,7 +12,7 @@ parquet (`data/online_listings_buy.parquet`) that the rest of the app consumes
|
|||
## How it works (and why it's careful about postcodes)
|
||||
|
||||
Every portal's **search** API exposes only an *outcode*-level address (e.g.
|
||||
`"…, London, SW9"`) plus map coordinates — never the full unit postcode. The
|
||||
`"…, London, SW9"`) plus map coordinates, never the full unit postcode. The
|
||||
full postcode lives on each listing's **detail page**, so the scraper fetches
|
||||
detail pages to recover it, and only trusts a detail postcode when its outcode
|
||||
agrees with the coordinate-nearest postcode (so a stale/wrong value can never
|
||||
|
|
@ -22,7 +22,7 @@ falls back to the coordinate-nearest postcode. See the module docstrings in
|
|||
|
||||
Detail fetching is the dominant cost, so it is:
|
||||
|
||||
- **cached across runs** — `data/detail_cache/{source}.json` maps listing id →
|
||||
- **cached across runs**: `data/detail_cache/{source}.json` maps listing id →
|
||||
recovered postcode; a re-run only fetches *newly-appeared* listings;
|
||||
- **fetched concurrently** for the HTTP portals (Rightmove, OnTheMarket), bounded
|
||||
by a shared global rate limiter so the VPN egress stays polite;
|
||||
|
|
@ -51,7 +51,7 @@ Also required: the ARCGIS postcode parquet at `../property-data/arcgis_data.parq
|
|||
|
||||
## Running
|
||||
|
||||
### Docker Compose (recommended — the only way that does Zoopla)
|
||||
### Docker Compose (recommended, the only way that does Zoopla)
|
||||
|
||||
`finder/docker-compose.yml` brings up the scraper plus **FlareSolverr** (which
|
||||
solves Zoopla's Cloudflare challenge), both sharing `media_gluetun`'s netns. This
|
||||
|
|
@ -106,18 +106,18 @@ GLUETUN_PROXY="" .venv/bin/python main.py --source onthemarket --outcodes SW9 \
|
|||
| Flag | Default | Meaning |
|
||||
|------|---------|---------|
|
||||
| `--source rightmove,onthemarket` | `all` | Comma-separated portal(s): any of `rightmove`, `onthemarket`, `zoopla`, or `all`. |
|
||||
| `--outcodes SW9,E14,BR1` | — | Specific outcodes (must be Greater-London-ish). Otherwise the full London set is loaded from ARCGIS. |
|
||||
| `--limit-outcodes N` | — | Cap the number of outcodes (quick smoke). |
|
||||
| `--max-properties-per-source N` | — | Stop each source after N transformed listings. |
|
||||
| `--outcodes SW9,E14,BR1` | none | Specific outcodes (must be Greater-London-ish). Otherwise the full London set is loaded from ARCGIS. |
|
||||
| `--limit-outcodes N` | none | Cap the number of outcodes (quick smoke). |
|
||||
| `--max-properties-per-source N` | none | Stop each source after N transformed listings. |
|
||||
| `--output-dir DIR` | `data/` | Where the parquet (and `detail_cache/`) are written. |
|
||||
| `--test` | off | ~10 likely-London outcodes, ≤100 listings/source, writes to `data/test/`. |
|
||||
|
||||
> **Always pass `--output-dir /tmp/...` for testing** — the default `data/` holds
|
||||
> **Always pass `--output-dir /tmp/...` for testing**: the default `data/` holds
|
||||
> the real listings the app consumes.
|
||||
|
||||
### Stopping a run
|
||||
|
||||
`Ctrl+C` (SIGINT) — or `docker stop` (SIGTERM) — triggers a **graceful
|
||||
`Ctrl+C` (SIGINT), or `docker stop` (SIGTERM), triggers a **graceful
|
||||
shutdown**: every source stops at its next outcode boundary, in-flight delays
|
||||
and retry backoffs wake immediately, and the run still persists the detail
|
||||
caches and writes the listings collected so far before exiting (code `130`).
|
||||
|
|
@ -147,7 +147,7 @@ A **separate enrich step** (outside `finder/`) turns that into
|
|||
`online_listings_buy_enriched.parquet`, which is what the Rust backend actually
|
||||
loads (`--actual-listings-path …/online_listings_buy_enriched.parquet` in the
|
||||
top-level `docker-compose.yml`). That enrich/scheduling pipeline is **not**
|
||||
documented here — only the raw scrape is.
|
||||
documented here. Only the raw scrape is documented.
|
||||
|
||||
The top-level `docker-compose.yml` (Rust `server`, `frontend`, `pocketbase`,
|
||||
`screenshot`) is the **web app**; it is downstream of the scrape and is **not**
|
||||
|
|
|
|||
|
|
@ -26,14 +26,14 @@ RETRY_BASE_DELAY = 2.0
|
|||
DETAIL_FETCH_CONCURRENCY = int(os.environ.get("DETAIL_FETCH_CONCURRENCY", "8"))
|
||||
REQUESTS_PER_SECOND = float(os.environ.get("REQUESTS_PER_SECOND", "10"))
|
||||
GRID_CELL_SIZE = 0.01 # degrees for postcode spatial index
|
||||
MAX_BEDROOMS = 20 # sanity cap — values above this are almost certainly parsing errors
|
||||
MAX_BEDROOMS = 20 # sanity cap: values above this are almost certainly parsing errors
|
||||
|
||||
TYPEAHEAD_URL = "https://los.rightmove.co.uk/typeahead"
|
||||
SEARCH_URL = "https://www.rightmove.co.uk/api/property-search/listing/search"
|
||||
RIGHTMOVE_BASE = "https://www.rightmove.co.uk"
|
||||
# Detail page (plain HTTPS GET, no Cloudflare). Its window.__PAGE_MODEL embeds
|
||||
# propertyData.address.{outcode,incode}, which together form the property's TRUE
|
||||
# full postcode — the search API only exposes the outcode. {id} is the numeric
|
||||
# full postcode. The search API only exposes the outcode. {id} is the numeric
|
||||
# listing id from the search response.
|
||||
RIGHTMOVE_DETAIL_URL = "https://www.rightmove.co.uk/properties/{id}"
|
||||
|
||||
|
|
@ -53,7 +53,7 @@ RIGHTMOVE_MAX_DETAILS_PER_OUTCODE = 4000 # max detail-page fetches per outcode
|
|||
# keep on APPROXIMATE pins (new-builds/developments) where Rightmove
|
||||
# deliberately fuzzes the coordinates. Degrades safely: when `pinType` is absent
|
||||
# from the search payload, nothing is skipped (behaviour is unchanged), so this
|
||||
# is only a speed-up to the extent the field is present — verify against a live
|
||||
# is only a speed-up to the extent the field is present. Verify against a live
|
||||
# search response before relying on the saving.
|
||||
RIGHTMOVE_SKIP_DETAILS_FOR_ACCURATE_PINS = (
|
||||
os.environ.get("RIGHTMOVE_SKIP_DETAILS_FOR_ACCURATE_PINS", "1") != "0"
|
||||
|
|
@ -94,7 +94,7 @@ GLUETUN_API_KEY = "My8AbvnKhfyFdRhpTVfoTfa5DkAMmg8K"
|
|||
GLUETUN_MAX_ROTATIONS = 0 # max egress-IP rotations per Cloudflare challenge
|
||||
|
||||
# Zoopla fetcher: "flaresolverr" (default) solves Cloudflare via the FlareSolverr
|
||||
# sidecar (docker-compose.yml) and needs no display/VNC — verified to return the
|
||||
# sidecar (docker-compose.yml) and needs no display/VNC, verified to return the
|
||||
# RSC flight stream with postcode + coordinates; "camoufox" drives a local
|
||||
# anti-fingerprint browser (needs an interactive solve on datacenter IPs).
|
||||
ZOOPLA_FETCHER = os.environ.get("ZOOPLA_FETCHER", "flaresolverr")
|
||||
|
|
@ -214,3 +214,16 @@ PROPERTY_TYPE_MAP = {
|
|||
CHANNELS = [
|
||||
{"channel": "BUY", "transactionType": "BUY", "sortType": "2"},
|
||||
]
|
||||
|
||||
# A second search pass that restricts the BUY channel to new-build developments
|
||||
# via Rightmove's `mustHave=newHome` filter, so new homes (which can rank low in
|
||||
# the default resale sort) are captured thoroughly. The API still echoes
|
||||
# `?channel=RES_BUY` in every listing URL regardless of this filter, so new
|
||||
# builds are identified by the per-listing `development` flag in
|
||||
# `transform_property`, which re-stamps the URL channel as RES_NEW.
|
||||
NEW_HOMES_CHANNEL = {
|
||||
"channel": "BUY",
|
||||
"transactionType": "BUY",
|
||||
"sortType": "2",
|
||||
"extra_params": {"mustHave": "newHome"},
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
"""FlareSolverr client — fetch Cloudflare-protected pages as rendered HTML.
|
||||
"""FlareSolverr client: fetch Cloudflare-protected pages as rendered HTML.
|
||||
|
||||
FlareSolverr (https://github.com/FlareSolverr/FlareSolverr) drives an
|
||||
undetected browser to pass Cloudflare's challenge and returns the fully
|
||||
|
|
@ -6,7 +6,7 @@ rendered HTML. It runs as a sidecar service (see docker-compose.yml) sharing
|
|||
the Gluetun VPN network namespace, so its browser egresses through the VPN.
|
||||
|
||||
Verified working against Zoopla's managed Turnstile on a datacenter VPN IP,
|
||||
provided a reused session and a generous maxTimeout (~120s) — the first
|
||||
provided a reused session and a generous maxTimeout (~120s): the first
|
||||
challenge solve is slow, subsequent requests on the warm session are fast.
|
||||
"""
|
||||
|
||||
|
|
|
|||
|
|
@ -132,7 +132,7 @@ def main() -> int:
|
|||
configure_standalone_runtime()
|
||||
configure_logging()
|
||||
# Ctrl+C (and SIGTERM, e.g. `docker stop`) asks the scrapers to wind down
|
||||
# gracefully — each source stops at its next outcode boundary and the run
|
||||
# gracefully. Each source stops at its next outcode boundary and the run
|
||||
# still persists detail caches and writes the listings collected so far.
|
||||
shutdown.install_signal_handlers()
|
||||
|
||||
|
|
|
|||
|
|
@ -1,4 +1,4 @@
|
|||
"""OnTheMarket (onthemarket.com) scraper — sale properties.
|
||||
"""OnTheMarket (onthemarket.com) scraper: sale properties.
|
||||
|
||||
OnTheMarket serves a Next.js app with the full search-results payload embedded
|
||||
as JSON in a `__NEXT_DATA__` script tag. No JS execution or browser needed:
|
||||
|
|
@ -15,19 +15,19 @@ Postcodes
|
|||
---------
|
||||
The search card exposes only an *outcode*-level address (e.g. "Padfield Road,
|
||||
London, SE5") and a map pin, so the old behaviour derived the postcode from the
|
||||
nearest postcode to that pin — a guess that frequently lands on a neighbouring
|
||||
nearest postcode to that pin, a guess that frequently lands on a neighbouring
|
||||
unit (the pin can sit on the wrong side of a street boundary).
|
||||
|
||||
Each *detail* page (`/details/{id}/`) is a plain HTTPS GET whose `__NEXT_DATA__`
|
||||
embeds the property's analytics dataLayer at
|
||||
`props.initialReduxState.metadata.dataLayer`, which carries the property's own
|
||||
`postcode` (full unit postcode, e.g. "SE5 9AA") keyed to this listing by
|
||||
`property-id`. Crucially this is NOT the agent's office postcode — that lives
|
||||
`property-id`. Crucially this is NOT the agent's office postcode. That lives
|
||||
separately at `…property.agent.postcode` ("SE5 8RS" for the same listing) and
|
||||
is the classic trap when blindly scanning the page for a postcode. We read the
|
||||
dataLayer postcode, verify `property-id` matches the listing, and accept it only
|
||||
when its outcode agrees with the coordinate-nearest postcode (via
|
||||
``resolve_listing_postcode``) — exactly the trust rule the other scrapers use.
|
||||
``resolve_listing_postcode``), exactly the trust rule the other scrapers use.
|
||||
Measured over a sample of real listings this yields a trustworthy, usually
|
||||
exact-unit postcode for ~11/12 listings; the rest safely fall back to the
|
||||
coordinate-nearest postcode.
|
||||
|
|
@ -68,8 +68,8 @@ from transform import (
|
|||
log = logging.getLogger("rightmove")
|
||||
|
||||
# Detail-page postcode recovery (see module docstring). When enabled, each
|
||||
# listing's detail page is fetched so its analytics dataLayer postcode — the
|
||||
# property's own full unit postcode — can replace the coordinate-nearest guess.
|
||||
# listing's detail page is fetched so its analytics dataLayer postcode (the
|
||||
# property's own full unit postcode) can replace the coordinate-nearest guess.
|
||||
# Bounded per outcode so a large outcode can't balloon into unbounded extra
|
||||
# HTTPS GETs. Kept at parity with the Rightmove/Zoopla detail caps (400) so a
|
||||
# typical outcode's listings all get their real postcode rather than a
|
||||
|
|
@ -145,7 +145,7 @@ def _fetch_page_json(client: httpx.Client, outcode: str, page_num: int) -> dict
|
|||
|
||||
if 300 <= resp.status_code < 400:
|
||||
log.debug(
|
||||
"OnTheMarket %s page %d redirected (%d) — end of results",
|
||||
"OnTheMarket %s page %d redirected (%d): end of results",
|
||||
outcode, page_num, resp.status_code,
|
||||
)
|
||||
return None
|
||||
|
|
@ -189,7 +189,7 @@ def parse_detail_postcode(html: str, listing_id: str | None = None) -> str | Non
|
|||
``props.initialReduxState.metadata.dataLayer.postcode`` and is the
|
||||
property's own unit postcode (e.g. "SE5 9AA"). It is deliberately NOT the
|
||||
agent's office postcode, which sits separately at
|
||||
``…property.agent.postcode`` — the trap when scanning a detail page for "a"
|
||||
``…property.agent.postcode``, the trap when scanning a detail page for "a"
|
||||
postcode. When ``listing_id`` is given, the dataLayer's ``property-id`` must
|
||||
match it, guaranteeing we read this listing's postcode and not a stray one.
|
||||
|
||||
|
|
@ -235,7 +235,7 @@ def _fetch_detail_postcode(
|
|||
|
||||
Results (including failures) are cached by listing id so a listing that
|
||||
reappears across overlapping outcode searches is fetched at most once. Plain
|
||||
HTTPS GET — OnTheMarket detail pages have no Cloudflare challenge. Network /
|
||||
HTTPS GET: OnTheMarket detail pages have no Cloudflare challenge. Network /
|
||||
parse errors degrade gracefully to None so the caller falls back to the
|
||||
coordinate-nearest postcode. Safe to call concurrently: distinct listing ids
|
||||
write distinct cache keys, and the shared RATE_LIMITER spaces the GETs.
|
||||
|
|
@ -452,7 +452,7 @@ def _prime_detail_postcodes(
|
|||
) -> None:
|
||||
"""Fill ``_detail_postcode_cache`` for the listings that need a detail page.
|
||||
|
||||
Picks the fresh (uncached) listings — up to ``detail_cap`` per outcode — then
|
||||
Picks the fresh (uncached) listings, up to ``detail_cap`` per outcode, then
|
||||
fetches their detail pages CONCURRENTLY, bounded by
|
||||
``DETAIL_FETCH_CONCURRENCY`` (the shared RATE_LIMITER keeps the combined
|
||||
request rate polite). Cached listings cost neither a slot nor a GET. The
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue