andras/perfect-postcode
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages 2 Wiki Activity Actions

Hide POIs, show overlay state, remove zoom button, rotate vpn

Browse source
This commit is contained in:
Andras Schmelczer 2026-07-15 21:58:05 +01:00
parent 35276d34fa
commit bce34b73de
32 changed files with 1137 additions and 186 deletions
Download patch file Download diff file Expand all files Collapse all files

6
finder/README.md
View file

@ -187,6 +187,9 @@ Environment variables (override the defaults in `constants.py`):
| `ZOOPLA_OUTCODE_TIMEOUT_SECONDS` | `300` | Per-outcode wall-clock budget for Zoopla. |
| `DETAIL_FETCH_CONCURRENCY` | `8` | Parallel detail fetches (Rightmove/OTM). |
| `REQUESTS_PER_SECOND` | `10` | Global request-rate cap. Lower it if you see `429`/`403`. |
| `BLOCK_403_THRESHOLD` | `5` | Consecutive 403s from one host before the egress IP is rotated. |
| `BLOCK_MAX_ROTATIONS` | `3` | Egress-IP rotations allowed per run. `0` disables rotation. |
| `MAX_ROW_DROP_RATIO` | `0.25` | Reject the run if the merged total falls this far below the previous parquet. |
| `RIGHTMOVE_SKIP_DETAILS_FOR_ACCURATE_PINS` | `1` | Inert today (see note above). |
Non-env code constants worth knowing (`constants.py` / `onthemarket.py`):
@ -216,7 +219,8 @@ uv run --with pytest pytest -q
| `scraper.py` | Orchestration: per-source runners, provider parallelism, cache load/save, merge + write. |
| `rightmove.py` / `onthemarket.py` / `zoopla.py` | Per-portal search + detail scraping and parsing. |
| `transform.py` | Raw listing → output schema; postcode trust rules. |
| `http_client.py` | Shared httpx client, retry/backoff, and the global `RATE_LIMITER`. |
| `http_client.py` | Shared httpx client, retry/backoff, the global `RATE_LIMITER`, and egress-block detection (`BlockedError`). |
| `gluetun.py` | Gluetun control-API client: reads the public IP and rotates the (shared) VPN tunnel. |
| `postcode_cache.py` | Persistent (cross-run) detail-cache load/save. |
| `spatial.py` | Grid spatial index for coordinate → nearest postcode. |
| `storage.py` | Parquet writer (server-ready column names). |