Work Lakes Area Graphix & Signworx
Auditing 600 product pages in twenty minutes
An agent pipeline that crawls a catalog, judges every product page against a rubric, rewrites what fails, and scores the rewrite before it counts as done. Most of the engineering went into not trusting the input — and into the failure modes that only show up once something runs unattended against the open internet.
- Role
- Sole author — architecture, crawler, prompts, evaluation harness
- Stack
- Node.js · Claude API · BullMQ · Redis · Cheerio · SerpAPI
- Links
- Source on GitHub
Scraped pages are untrusted input, and they reach a model
Every page the crawler returns is text somebody else controls. Once that text is inside a prompt, an instruction sitting in a product description is competing with mine. So scraped content never enters the instruction channel: a sanitizer strips known injection patterns and collapses repeated-token floods, and whatever survives is wrapped in an explicit boundary that tells the model to treat the payload strictly as data.
A dedicated test suite holds that behaviour in place, so a future refactor cannot quietly remove it.
A crawler that accepts URLs is an SSRF hole
A scraper that fetches whatever URL it is handed can be pointed at the private network it runs inside; cloud metadata endpoints are the classic target. Every hostname is resolved before the request is made and refused if it lands on a private, loopback, or link-local range, so a redirect to 169.254.169.254 fails instead of turning the crawler into a request proxy.
robots.txt is fetched, cached per host, and honored.
crawl guard — resolution order
“The model returned something” is not success
A run that finishes without errors can still be a run that quietly made six hundred pages worse. Generated output is scored against explicit rules — required sections present, and generic filler headings like “Overview” or “Product Details” rejected outright. A page that fails is regenerated rather than shipped.
That retry is capped at two attempts. Unbounded regeneration turns one bad page into an expensive loop, so a page that genuinely cannot pass fails loudly instead of burning tokens in the dark.
Long runs die in the middle
Work moves through a job queue with one worker per pipeline stage, and stage completion is the checkpoint. Interrupting a six-hundred-page run and restarting it resumes from the last finished stage per page rather than re-crawling and re-billing everything.
Retries are limited to genuinely retryable failures — 429, 502, 503, 504, timeouts, connection resets — with exponential backoff and jitter. A malformed request is not retried, because retrying it only fails again more expensively.
What it moved
- AI search impressions, up from ~300K
- 5.9M
- Product pages audited and optimized
- 800+
- Wall clock for a 600-page pass
- ~20 min