Scraping stack for the fleet — spike-tested comparison

Summary

• The Hermes skill page Young linked covers Scrapling (D4Vinci/Scrapling), not crawl4ai/scrapy.
• Repo-liveness gate passed both: crawl4ai 85,665 stars / Apache-2.0 / pushed 2026-10-15; scrapy 64,858 stars / BSD-3-Clause / pushed 2026-10-16. Neither archived.
• Hands-on spike (scratch venv, Python 3.14, real fetch of quotes.toscrape.com): scrapy 2.19.0 works out of the box (10 quotes, 1.3s); scrapling 0.4.15 needs pip install "scrapling[all]" — the bare install is broken (missing curl_cffi, patchright, browserforge in dependency order), then 10 quotes in 1.0s.
• All three require Python >= 3.10 — compatible with fleet Python 3.14.7.

Key findings

1. Scrapling skill is not installed in any fleet profile. The Hermes-blessed path (hermes skills install official/research/scrapling) is the only one of the three with stealth/Cloudflare-bypass + spider + CLI in a single package. Install it first.
2. scrapling 0.4.15 packaging is fragile. Bare pip install scrapling fails at import (three missing transitive deps); the [all] extra + 120MB Playwright chromium download fixes it. Record the full install line, never the bare one.
3. scrapy is the bulk-pipeline choice. No browser needed for static pages, Zyte-maintained, BSD license with no attribution clause. Fits Eva's scheduled collectors (ASX scans, market watch) as-is.
4. crawl4ai is the agent-consumption choice. Its output is LLM-ready Markdown with citations out of the box — exactly what research verification consumes. Heaviest install. License note: Apache-2.0 with an attribution requirement (badge or text line) — if we ship anything built on it, the attribution must travel with it.
5. No single winner. Skill (stealth one-offs) + scrapy (scheduled bulk) + crawl4ai trial (agent-facing extraction) covers every fleet scraping need with no overlap.

Implications for us

• Install the scrapling skill fleet-wide (one command, covers the Cloudflare cases web_extract can't reach).
• Eva's collectors stay on scrapy patterns — no rewrite needed.
• Trial crawl4ai for my research-verification lane where clean Markdown output saves a conversion step.
• The spike venv lives at /tmp/scrape_spike (scratch, will be pruned) — reinstall from the pinned versions above, not from memory.

Sources

• Skill definition: hermes-agent docs, research-scrapling page (Scrapling v1.0.0, MIT, FEUAZUR)
• crawl4ai README (main branch): v0.9.5 on PyPI, features verified from README prose (Docker/MCP/crash-recovery not spike-tested)
• scrapy README (master): v2.19.0 on PyPI, maintained by Zyte
• Spike runs: local transcripts 2026-10-17 (scrapy 1.3s / scrapling 1.0s on identical target)