← All notes

A scraper that repairs itself

Watching a web page is easy once. You find the element that holds the price, you write down how to get to it, and you check it on a schedule.

Then the site ships a redesign and your selector points at nothing. That is the actual problem in web monitoring, and it is the one worth designing around.

The lazy answer costs too much

The obvious fix is to stop writing selectors at all. Send the page to a language model on every run and ask what the price is.

That works, and it is expensive in two ways. You pay tokens on every single check, forever, on a page that mostly has not changed. And the model can be inconsistent between runs in ways a selector never is, so you inherit a small amount of noise as a permanent feature.

The better shape is to treat the model as a repair mechanic rather than a meter reader.

Three tiers in order. Structured data and a cached selector each cost zero model tokens; only when both miss does a field reach the repair tier, which costs five to twenty thousand input tokens

Three tiers, cheapest first

Structured data. A lot of sites already publish machine-readable facts about themselves. Product pages carry application/ld+json with a price and a stock status, articles carry a publish date, listing pages carry an item list. Meta tags carry more of the same.

When the value you want is in there, the thing to remember is not a CSS selector at all. It is a path into that JSON:

ld:Product/offers/price
ld:ItemList/itemListElement/*/name
meta:product:price:amount

Those survive a redesign. The visual layer can be rebuilt entirely and the schema block usually comes along unchanged, because it is generated from the same database row the page is. This is the fast path worth reaching for first, and on a typical product page it answers the question on the very first check for zero tokens.

A cached selector. If structured data does not have it, fall back to the CSS selector learned last time. Read it, coerce it to the type the field is supposed to be, and check that the result is plausible. A selector that returns nothing, or returns text where a number belongs, is treated as broken rather than trusted.

Repair. Only the fields that are missing or broken go to the model.

Condensing is most of the work

You do not send raw HTML. You send a numbered list of the visible elements:

[12] h1 Sony WH-1000XM5 Wireless Headphones
[13] span $348.00
[14] a https://example.com/reviews Reviews (2,314)

Scripts, styles, navigation, footers, forms and anything hidden are dropped. If the page has a <main>, only that subtree is used. What is left is a fraction of the original size and contains almost all of the meaning.

Every listed element also gets a unique CSS selector computed for it ahead of time, preferring stable handles: an id, a data-testid, an itemprop, a real class name. Hashed and utility class names are skipped, because a selector built on css-1x2y3z or a stack of Tailwind atoms is a selector that breaks on the next build.

The model is then asked a narrow question: which numbered elements hold each field, and how confident are you.

The gate that makes it safe

Here is the part that matters, and it is one line of logic.

A selector the model suggests is accepted only if reading that selector actually reproduces the value the model said it found.

Without that check you are caching the model's guess. With it, you are caching something you verified against the page in front of you. A confident wrong answer fails the check and gets discarded instead of being written down as truth and replayed on every future run.

Lists work the same way with one extra step. Take the paths of the elements the model cited, find the segment where they diverge, and drop the positional part at exactly that segment. What was a path to the third item becomes a path to every item, including ones that do not exist yet. Accept it only if it returns at least as many items as the model found.

What the cost curve looks like

Steady state is free. Cached selectors and structured data cost a fetch and a few hundred milliseconds of parsing.

Repair costs roughly the size of the condensed page in input tokens, typically five to twenty thousand, and a few hundred out. That happens when a site redesigns, which for most sites is rarely.

So the spend is not proportional to how often you check. It is proportional to how often the world changes underneath you, which is the correct thing to be paying for.

What it does not fix

Pages behind a login or a paywall are out of scope on purpose. Infinite scroll is read as far as the first render goes. Selectors generalise across siblings on one page but not across pagination. And sites with serious bot protection will still say no, which is their right.

Self-healing is not the same as unbreakable. It just means the common break is handled by the system instead of by you, at two in the morning, wondering why the alert went quiet.