andras/perfect-postcode
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages 2 Wiki Activity Actions

..

Browse source
This commit is contained in:
Andras Schmelczer 2026-07-03 18:47:28 +01:00
parent ab688243d7
commit 463bd4c647
54 changed files with 13239 additions and 625 deletions
Download patch file Download diff file Expand all files Collapse all files

22
analyses/source_overlap.ipynb
View file

@ -5,7 +5,7 @@
"id": "db1n423kpm8",
"metadata": {},
"source": [
"# Rightmove vs Home.co.uk — Source Overlap Analysis\n",
"# Rightmove vs Home.co.uk: Source Overlap Analysis\n",
"\n",
"The property scraper collects listings from two sources: **Rightmove** and **home.co.uk**. During merging, cross-source deduplication removes home.co.uk listings that match a Rightmove listing by `(postcode, bedrooms, price)`.\n",
"\n",
@ -81,7 +81,7 @@
"\n",
"The merged parquet contains already-deduplicated data. Home.co.uk listings that matched a Rightmove listing by `(postcode, bedrooms, price)` were removed during scraping. The log reported **2,220 cross-source dedupes** for BUY.\n",
"\n",
"So the true home.co.uk total was 20,650 (unique) + 2,220 (deduped) = **22,870** — giving a **9.7% overlap rate** on the outcodes that were scraped."
"So the true home.co.uk total was 20,650 (unique) + 2,220 (deduped) = **22,870**, giving a **9.7% overlap rate** on the outcodes that were scraped."
]
},
{
@ -3164,7 +3164,7 @@
"---\n",
"## 3. Approximate Overlap via Fuzzy Matching\n",
"\n",
"The scraper deduped by exact `(postcode, bedrooms, price)`. We can also check for near-matches — properties at the same postcode with similar price that might be the same listing with slightly different data."
"The scraper deduped by exact `(postcode, bedrooms, price)`. We can also check for near-matches: properties at the same postcode with similar price that might be the same listing with slightly different data."
]
},
{
@ -5246,7 +5246,7 @@
}
],
"source": [
"# Property sub-type comparison — top home.co.uk sub-types\n",
"# Property sub-type comparison: top home.co.uk sub-types\n",
"hk_subtypes = (\n",
" buy.filter(pl.col(\"source\") == \"Home.co.uk\")[\"Property sub-type\"]\n",
" .value_counts()\n",
@ -6304,7 +6304,7 @@
"---\n",
"## 7. What Does Home.co.uk Add?\n",
"\n",
"Home.co.uk listings that passed the dedup filter are genuinely unique — not on Rightmove at all (or listed with different price/beds). What do they look like?"
"Home.co.uk listings that passed the dedup filter are genuinely unique: not on Rightmove at all (or listed with different price/beds). What do they look like?"
]
},
{
@ -6409,13 +6409,13 @@
"output_type": "stream",
"text": [
"\n",
"Rightmove — days on market:\n",
"Rightmove: days on market:\n",
" Median: 74\n",
" Mean: 140\n",
" P25: 26\n",
" P75: 189\n",
"\n",
"Home.co.uk — days on market:\n",
"Home.co.uk: days on market:\n",
" Median: 156\n",
" Mean: 164\n",
" P25: 50\n",
@ -6424,7 +6424,7 @@
}
],
"source": [
"# Listing age comparison — are home.co.uk listings older/newer?\n",
"# Listing age comparison: are home.co.uk listings older/newer?\n",
"import datetime\n",
"\n",
"now = datetime.datetime(2026, 3, 11)\n",
@ -6435,7 +6435,7 @@
"for src in [\"Rightmove\", \"Home.co.uk\"]:\n",
" age = with_age.filter(pl.col(\"source\") == src)[\"days_on_market\"].drop_nulls()\n",
" if len(age) > 0:\n",
" print(f\"\\n{src} — days on market:\")\n",
" print(f\"\\n{src}: days on market:\")\n",
" print(f\" Median: {age.median():.0f}\")\n",
" print(f\" Mean: {age.mean():.0f}\")\n",
" print(f\" P25: {age.quantile(0.25):.0f}\")\n",
@ -7388,7 +7388,7 @@
" Projected unique additions: ~274,584\n",
" Projected merged dataset: ~728,899 (60.4% increase)\n",
"\n",
"⚠️ These are rough estimates — the covered outcodes may not be representative\n"
"⚠️ These are rough estimates: the covered outcodes may not be representative\n"
]
}
],
@ -7425,7 +7425,7 @@
" f\" Projected merged dataset: ~{rm_buy + projected_unique:,} ({projected_unique / rm_buy * 100:.1f}% increase)\"\n",
")\n",
"print()\n",
"print(\"⚠️ These are rough estimates — the covered outcodes may not be representative\")"
"print(\"⚠️ These are rough estimates: the covered outcodes may not be representative\")"
]
},
{