{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "7fb27b941602401d91542211134fc71a",
   "metadata": {},
   "source": [
    "# Face-aware crop vs. centre crop\n",
    "\n",
    "`face_aware_crop` (in `src/lib/crop.py`) resizes-to-cover then biases the crop toward\n",
    "Immich-detected face boxes — keeping every face in frame when they fit, falling back to an\n",
    "area-weighted centroid when they don't, and centre-cropping when there are no faces.\n",
    "\n",
    "This notebook scans a pool of photos, fetches their face boxes, and picks the cases where\n",
    "the face-aware crop is **maximally different from a plain centre crop** — those are the ones\n",
    "where the smart logic earns its keep. For each pick it shows:\n",
    "\n",
    "1. Original, with face boxes drawn (head-extended boxes in cyan, raw boxes in yellow)\n",
    "2. Plain centre crop result (what `_crop_center` would do)\n",
    "3. Face-aware crop result (what production uses)\n",
    "4. Crop-window overlay on the original (centre = orange, face-aware = green)\n",
    "\n",
    "Plus one **no-faces** case to confirm the fallback behaves like a plain centre crop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "acae54e37e7d407bbb7b55eff062a284",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "\n",
    "sys.path.insert(0, \".\")\n",
    "from _helpers import (\n",
    "    CACHE_DIR,\n",
    "    bootstrap,\n",
    "    download_image,\n",
    "    fetch_pool,\n",
    "    immich_client,\n",
    "    show_grid,\n",
    "    silenced,\n",
    ")\n",
    "\n",
    "bootstrap()\n",
    "\n",
    "import json\n",
    "import math\n",
    "\n",
    "import numpy as np\n",
    "from crop import HEAD_EXTENSION, face_aware_crop\n",
    "from PIL import Image, ImageDraw\n",
    "from waveshare_epd.epd7in3e import EPD_HEIGHT, EPD_WIDTH, _crop_center\n",
    "\n",
    "POOL_SIZE = 80  # smaller than auto-tune notebook — each photo costs an extra API call for faces\n",
    "SEED = 7\n",
    "N_PICKS = 4  # most-divergent picks (plus one no-faces baseline)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9a63283cbaf04dbcab1f6479b197f3a8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pool size: 80 landscape photos\n",
      "face data: 80 fetched, 0 cached\n"
     ]
    }
   ],
   "source": [
    "client = immich_client()\n",
    "pool_assets = fetch_pool(client, pool_size=POOL_SIZE, seed=SEED)\n",
    "print(f\"pool size: {len(pool_assets)} landscape photos\")\n",
    "\n",
    "# Cache the face lookups — each is a separate /assets/{id} API call.\n",
    "face_cache_path = CACHE_DIR / \"faces.json\"\n",
    "CACHE_DIR.mkdir(exist_ok=True)\n",
    "face_cache = json.loads(face_cache_path.read_text()) if face_cache_path.exists() else {}\n",
    "\n",
    "fetched_now = 0\n",
    "for asset in pool_assets:\n",
    "    if asset[\"id\"] in face_cache:\n",
    "        continue\n",
    "    face_cache[asset[\"id\"]] = client.get_asset_faces(asset[\"id\"])\n",
    "    fetched_now += 1\n",
    "if fetched_now:\n",
    "    face_cache_path.write_text(json.dumps(face_cache))\n",
    "print(f\"face data: {fetched_now} fetched, {len(pool_assets) - fetched_now} cached\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8dd0d8092fe74a7c96281538738b07e2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "80 with faces, 0 without\n",
      "crop shift (px): min=1  median=45  max=60\n"
     ]
    }
   ],
   "source": [
    "# For each asset, compute centre-crop offset and face-aware-crop offset on the\n",
    "# resize-to-cover canvas. We re-implement just the offset math here so the comparison\n",
    "# doesn't depend on actually invoking the crops yet.\n",
    "def crop_offsets(img_w, img_h, target_w, target_h, faces):\n",
    "    img_aspect = img_w / img_h\n",
    "    target_aspect = target_w / target_h\n",
    "    if img_aspect < target_aspect:\n",
    "        new_w, new_h = target_w, math.ceil(target_w / img_aspect)\n",
    "    else:\n",
    "        new_w, new_h = math.ceil(target_h * img_aspect), target_h\n",
    "\n",
    "    centre_off = (max(0, (new_w - target_w) // 2), max(0, (new_h - target_h) // 2))\n",
    "    cx, cy = new_w / 2, new_h / 2\n",
    "    if faces:\n",
    "        boxes = []\n",
    "        for f in faces:\n",
    "            sx = new_w / (f.get(\"imageWidth\") or img_w)\n",
    "            sy = new_h / (f.get(\"imageHeight\") or img_h)\n",
    "            x1 = f[\"boundingBoxX1\"] * sx\n",
    "            y1 = f[\"boundingBoxY1\"] * sy\n",
    "            x2 = f[\"boundingBoxX2\"] * sx\n",
    "            y2 = f[\"boundingBoxY2\"] * sy\n",
    "            boxes.append((x1, y1, x2, y2, max(0.0, (x2 - x1) * (y2 - y1))))\n",
    "        x_lo, x_hi = min(b[0] for b in boxes), max(b[2] for b in boxes)\n",
    "        if x_hi - x_lo <= target_w:\n",
    "            cx = (x_lo + x_hi) / 2\n",
    "        else:\n",
    "            total = sum(b[4] for b in boxes) or 1.0\n",
    "            cx = sum((b[0] + b[2]) / 2 * b[4] for b in boxes) / total\n",
    "        y_lo_ext = min(b[1] - (b[3] - b[1]) * HEAD_EXTENSION for b in boxes)\n",
    "        y_hi = max(b[3] for b in boxes)\n",
    "        if y_hi - y_lo_ext <= target_h:\n",
    "            cy = (y_lo_ext + y_hi) / 2\n",
    "        else:\n",
    "            total = sum(b[4] for b in boxes) or 1.0\n",
    "            cy = sum((b[1] + b[3]) / 2 * b[4] for b in boxes) / total\n",
    "    face_off = (\n",
    "        max(0, min(int(cx - target_w / 2), new_w - target_w)),\n",
    "        max(0, min(int(cy - target_h / 2), new_h - target_h)),\n",
    "    )\n",
    "    return centre_off, face_off, (new_w, new_h), boxes if faces else []\n",
    "\n",
    "\n",
    "stats = []\n",
    "for asset in pool_assets:\n",
    "    exif = asset.get(\"exifInfo\") or {}\n",
    "    iw, ih = exif.get(\"exifImageWidth\") or 0, exif.get(\"exifImageHeight\") or 0\n",
    "    if exif.get(\"orientation\") in (6, 8, \"6\", \"8\"):\n",
    "        iw, ih = ih, iw\n",
    "    if not (iw and ih):\n",
    "        continue\n",
    "    faces = face_cache.get(asset[\"id\"], [])\n",
    "    centre_off, face_off, canvas, boxes = crop_offsets(iw, ih, EPD_WIDTH, EPD_HEIGHT, faces)\n",
    "    dx = face_off[0] - centre_off[0]\n",
    "    dy = face_off[1] - centre_off[1]\n",
    "    stats.append(\n",
    "        {\n",
    "            \"asset\": asset,\n",
    "            \"faces\": faces,\n",
    "            \"boxes\": boxes,\n",
    "            \"centre_off\": centre_off,\n",
    "            \"face_off\": face_off,\n",
    "            \"canvas\": canvas,\n",
    "            \"shift\": math.hypot(dx, dy),\n",
    "            \"dx\": dx,\n",
    "            \"dy\": dy,\n",
    "        }\n",
    "    )\n",
    "\n",
    "with_faces = [s for s in stats if s[\"faces\"]]\n",
    "no_faces = [s for s in stats if not s[\"faces\"]]\n",
    "print(f\"{len(with_faces)} with faces, {len(no_faces)} without\")\n",
    "shifts = np.array([s[\"shift\"] for s in with_faces])\n",
    "if len(shifts):\n",
    "    print(\n",
    "        f\"crop shift (px): min={shifts.min():.0f}  median={np.median(shifts):.0f}  max={shifts.max():.0f}\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "72eea5119410473aa328ad9291626812",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name                                  faces     dx    dy  |shift|\n",
      "IMG_6698.JPG                              2     +0   -60       60\n",
      "IMG_2373.JPG                              2     +0   -60       60\n",
      "3d91beaa-9136-4488-bc17-c0c333fd79bb      5     +0   -60       60\n",
      "IMG_2538.JPG                              2     +0   -60       60\n"
     ]
    }
   ],
   "source": [
    "with_faces.sort(key=lambda s: s[\"shift\"], reverse=True)\n",
    "picks = with_faces[:N_PICKS]\n",
    "if no_faces:\n",
    "    picks.append(no_faces[0])\n",
    "\n",
    "print(f\"{'name':36s}  {'faces':>5s}  {'dx':>5s} {'dy':>5s}  {'|shift|':>7s}\")\n",
    "for s in picks:\n",
    "    name = (s[\"asset\"].get(\"originalFileName\") or s[\"asset\"][\"id\"])[:36]\n",
    "    print(\n",
    "        f\"{name:36s}  {len(s['faces']):>5d}  {s['dx']:>+5.0f} {s['dy']:>+5.0f}  {s['shift']:>7.0f}\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "8edb47106e1a46a883d545849b8ab81b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Center cropping...\n",
      "Center cropping...\n",
      "Center cropping...\n",
      "Center cropping...\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 3597316 characters]",
      "text/plain": [
       "<Figure size 2000x1280 with 16 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "def draw_boxes_on_canvas(image, faces, target=(EPD_WIDTH, EPD_HEIGHT)):\n",
    "    \"\"\"Resize-to-cover the image, draw face boxes on it (cyan = head-extended, yellow = raw).\"\"\"\n",
    "    iw, ih = image.size\n",
    "    img_aspect = iw / ih\n",
    "    target_aspect = target[0] / target[1]\n",
    "    if img_aspect < target_aspect:\n",
    "        new_w, new_h = target[0], math.ceil(target[0] / img_aspect)\n",
    "    else:\n",
    "        new_w, new_h = math.ceil(target[1] * img_aspect), target[1]\n",
    "    canvas = image.resize((new_w, new_h), Image.LANCZOS).copy()\n",
    "    draw = ImageDraw.Draw(canvas)\n",
    "    for f in faces:\n",
    "        sx = new_w / (f.get(\"imageWidth\") or iw)\n",
    "        sy = new_h / (f.get(\"imageHeight\") or ih)\n",
    "        x1, y1 = f[\"boundingBoxX1\"] * sx, f[\"boundingBoxY1\"] * sy\n",
    "        x2, y2 = f[\"boundingBoxX2\"] * sx, f[\"boundingBoxY2\"] * sy\n",
    "        head_y1 = y1 - (y2 - y1) * HEAD_EXTENSION\n",
    "        draw.rectangle([x1, head_y1, x2, y2], outline=(0, 220, 255), width=4)\n",
    "        draw.rectangle([x1, y1, x2, y2], outline=(255, 220, 0), width=2)\n",
    "    return canvas, (new_w, new_h)\n",
    "\n",
    "\n",
    "def draw_crop_windows(canvas, centre_off, face_off, target=(EPD_WIDTH, EPD_HEIGHT)):\n",
    "    out = canvas.copy()\n",
    "    draw = ImageDraw.Draw(out)\n",
    "    cx, cy = centre_off\n",
    "    fx, fy = face_off\n",
    "    draw.rectangle([cx, cy, cx + target[0], cy + target[1]], outline=(255, 140, 0), width=4)\n",
    "    draw.rectangle([fx, fy, fx + target[0], fy + target[1]], outline=(0, 200, 0), width=4)\n",
    "    return out\n",
    "\n",
    "\n",
    "rows, titles = [], []\n",
    "for s in picks:\n",
    "    img = download_image(client, s[\"asset\"])\n",
    "    canvas, _ = draw_boxes_on_canvas(img, s[\"faces\"])\n",
    "    windowed = draw_crop_windows(canvas, s[\"centre_off\"], s[\"face_off\"])\n",
    "    centre = _crop_center(img, EPD_WIDTH, EPD_HEIGHT)\n",
    "    smart = face_aware_crop(img, EPD_WIDTH, EPD_HEIGHT, s[\"faces\"])\n",
    "    name = (s[\"asset\"].get(\"originalFileName\") or s[\"asset\"][\"id\"])[:24]\n",
    "    rows.append([canvas, centre, smart, windowed])\n",
    "    titles.append(\n",
    "        [\n",
    "            f\"{name}\\n{len(s['faces'])} face{'' if len(s['faces']) == 1 else 's'}\",\n",
    "            \"centre crop\",\n",
    "            f\"face-aware crop\\nshift: ({s['dx']:+.0f}, {s['dy']:+.0f}) px\",\n",
    "            \"crop windows on canvas\\norange = centre, green = face-aware\",\n",
    "        ]\n",
    "    )\n",
    "\n",
    "with silenced():\n",
    "    show_grid(rows, titles, figsize_scale=(5.0, 3.2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10185d26023b46108eb7d9f57d49d2b3",
   "metadata": {},
   "source": [
    "## Reading the comparison\n",
    "\n",
    "- **Cyan boxes** are the head-extended face regions the algorithm tries to keep in frame —\n",
    "  the extension (`HEAD_EXTENSION`) accounts for the fact that Immich's boxes end at the hairline.\n",
    "- **Yellow boxes** are the raw bounding boxes Immich reported.\n",
    "- The biggest visual win is when faces are clustered to one side, near a horizontal or vertical\n",
    "  edge: centre crop slices through them, face-aware crop slides the window to keep them whole.\n",
    "- When face span exceeds the crop dimension on an axis, the algorithm falls back to an\n",
    "  area-weighted centroid — that's intentional: it biases toward the foreground (largest) face\n",
    "  rather than splitting the crop badly between far-apart subjects.\n",
    "- The no-faces row should look identical between centre and face-aware crops — that confirms\n",
    "  the no-face fallback path."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "frame (3.12.13)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
