From the workshop. This is how the engine behind Guardian Bots works, a tool we are building that watches web pages for the things people check by hand: a price, a listing, a job posting, an appointment slot. It has not launched. Every step below is in its code, and its 48 engine tests pass.
Tools that watch web pages exist. They all fail the same way: a site changes its HTML, the selector breaks, and somebody has to go and fix it. That maintenance is why web monitoring has stayed unreliable. The idea here is a watcher that repairs itself, and the interesting part is how little it needs the expensive repair.
1. Fetch the page politely
The worker checks the site's robots.txt, and waits its turn behind any other fetch to the same host. It refuses private and internal network addresses, and checks again on every redirect, because a public page that redirects to an internal address would otherwise walk straight past the first check. It tries a plain request first, and only if the answer is a JavaScript shell with no content does it load the page in a headless browser. Images, fonts and media are never downloaded.
2. Read what the page says about itself
Most shopping, listing, job and article pages publish their own structured data: schema.org JSON-LD in a script tag, the same vocabulary written into the markup as microdata, or Open Graph tags. A product page will often state its price, currency and availability in a form meant for machines to read.
So that is read first. When a field the watcher wants ("price", "in stock", "job title") clearly maps onto one of those, it is read straight from the page's own declaration, with no guessing. Those readings survive any redesign, because a site that changes its layout rarely changes what it tells search engines.
3. Try the cached selector, and do not trust it blindly
For everything else, each field keeps a CSS selector from the last time the watcher knew where the value lived. The value is read, converted to the declared type (currency, number, yes or no, list) and checked: a price that parses to a number is plausible, an empty string or a "Loading..." placeholder is not.
The dangerous failure is not the selector that breaks. It is the selector that still matches after a redesign, but now points at a different element. The classic case is a price selector that lands on the struck-through former price, which would fire a false "price dropped" alert. So when the page's own structured data disagrees with what a selector read, the structured data wins and the selector is learned again. The engine's tests include exactly that page.
4. Repair the rest with a model
Only when a field still has no plausible value does a language model get involved. The page is condensed into a numbered list of its visible elements: tag, text, and a link if there is one, with scripts and hidden elements dropped and the structured data listed first. The model gets that list and the field descriptions, and returns each value together with the numbers of the elements it read it from.
Those element numbers map back to selectors. Each learned selector is checked to reproduce the value on the same page before it is kept, and for a list it is generalised across the cited items so it also matches the next ones. On the next check the cheap path works again.
Selectors are fast and free but brittle. A model is robust but costs time and money. Using the cheap reading by default and the model only for repair gets the reliability of one at close to the price of the other, and every repair teaches the watcher where the data moved, so the expensive path gets rarer.
5. Tell you once
A condition like "the price is below a number" stays true for as long as the page stays that way. Firing on every check while it holds would mean a watcher that checks every few hours emailing you several times a day until you unsubscribe, which is worse than not alerting at all.
So conditions fire on the edge: the check on which the condition became true, and not again until it goes false and comes back. Only successful checks move the baseline, so a failed check in between never re-arms an alert that has already been sent. A watcher for changes records its first check as the baseline and says nothing, since there is nothing yet to compare against.
What this has in common with everything else here
The same engine's fetcher and its structured-data-first reading now power the research fetcher this site's own tools use, minus anything that hides what it is. A block page is reported as a block, never treated as a puzzle. The shape is the same as the rest of this blog: read what the source says about itself first, check the fast answer against a second route, and only then reach for the expensive one.