andras/perfect-postcode
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages 2 Wiki Activity Actions

..

Browse source
This commit is contained in:
Andras Schmelczer 2026-07-03 18:39:34 +01:00
parent 1ee796b282
commit ab688243d7
36 changed files with 307 additions and 135 deletions
Download patch file Download diff file Expand all files Collapse all files

8
pipeline/download/gias.py
View file

@ -4,12 +4,12 @@ GIAS is the DfE register of all educational establishments in England, updated
daily. The CSV is generated on-demand via a four-step interaction with the
public Downloads page (there is no static URL):
1. GET /Downloads — extract anti-forgery token, the `all.edubase.data` tag,
1. GET /Downloads: extract anti-forgery token, the `all.edubase.data` tag,
and the FileGeneratedDate that the server expects for that tag today.
2. POST /Downloads/Collate — submit the form to start file generation. The
2. POST /Downloads/Collate: submit the form to start file generation. The
redirect URL contains a generation UUID.
3. Poll /Downloads/GenerateAjax/{id} until status:true.
4. GET the Azure blob URL with ?id={id} — returns a ZIP containing
4. GET the Azure blob URL with ?id={id}: returns a ZIP containing
`edubasealldataYYYYMMDD.csv`.
The CSV is cp1252-encoded with 135 columns. We keep the fields useful for a
@ -262,7 +262,7 @@ def transform(zip_bytes: bytes) -> pl.DataFrame:
.cast(pl.Float32, strict=False),
)
# Drop rows without coordinates — a small number of historic/dummy entries
# Drop rows without coordinates: a small number of historic/dummy entries
# have Easting=0 which would map to the Atlantic.
df = df.filter(
pl.col("Easting").is_not_null()