pipeline / brief → discover → crawl → assess → steady
Agents do the judgement — what a domain is, where its sources are, what on a page matters. Code does everything that can be checked: validation, deduplication, change detection, caps. The result is a graph that builds itself from one sentence and stays current for the price of what actually changed.
One sentence becomes a structured brief: the domain's name, language and region; what a person, object, location and event mean here; inclusion and exclusion rules; real-world examples a complete graph must contain; seed search queries per kind; and URL patterns to avoid. Code turns the brief into the domain's published gate.
Seed queries run against web search in the domain's language. Every candidate source is validated in code — crawlable URL, allowed host, robots.txt, reachable HTML, a per-host quota — before it becomes a source.
The pipeline fetches pages politely, converts each to markdown, and extracts POLE entities, facets, relationships and the next links worth following. The agent, not a link-follower, decides where to go next.
Deterministic hygiene removes junk names and merges duplicates. A precision sweep checks every entity against the brief. Locations are geocoded. A judge scores coverage and precision and proposes deletes, merges, gate refinements and gap searches — which code validates and applies, capped so one bad verdict cannot wipe a graph.
After a few cycles the domain settles: scheduled re-scrapes, a weekly frontier plan, daily sweeps of newly arrived entities, periodic re-assessment. A watchdog resumes any stage that stalls.
Every domain has a published gate: plain-language inclusion and exclusion rules plus URL patterns. The extraction agent applies it page by page — “include firms whose main business is IT consulting with an office in Linköping or Norrköping; exclude recruitment ads” — and an entity the exclusion rules cover is not extracted even when the page is relevant. Gates are versioned; the judge's refinements become new versions, and a widened gate gives previously rejected pages a second look.
No HTML ever reaches a model. Each page is converted to markdown with navigation, footers and scripts removed and links kept inline. The extraction model returns a structured verdict: is the page relevant, how confident is it, is it a catalog, an article or an entity's own page — and the entities, facets, relationships and up to ten links worth following. A shaky verdict, especially a shaky rejection, is re-read by a stronger model.
Every entity and relationship is stored with the page and a verbatim snippet it came from. Relationships are normalized to 15 canonical predicates, and names are deduplicated against the graph — including by canonical URL, so two names for the same official site become one entity with an alias.
Re-scraping is where knowledge graphs usually get expensive. Nolvin re-reads every page in tiers and stops at the first one that answers:
In production, diff reads cost about a third of a full read, and unchanged pages cost nothing — keeping all of Nolvin's live graphs current runs well under a dollar a day.
Graphs built from the web drift toward noise: navigation labels, near-duplicates, adjacent topics. Each autopilot cycle cleans before it grows:
Once built, a domain maintains itself. The scheduler requeues due pages every six hours, a frontier planner ranks unexplored links weekly and runs gap searches, new entities are swept daily, and entities that lose all support expire after 21 days. Webhooks and a server-sent event stream report what changed.
Page extraction runs on Claude Haiku 4.5, a fast and inexpensive model; pages where it is unsure are re-read by Claude Opus. Judgement-heavy steps — the domain brief, source discovery, the frontier planner and the graph judge — run on Claude Opus. Every model call is metered per domain.
Yes. NolvinBot reads robots.txt before crawling a host (cached for 24 hours), honors Allow, Disallow and Crawl-delay, and spaces requests to the same host — at least 1.5 seconds apart by default, longer if the site asks. See NolvinBot.
A scheduler runs every six hours. Catalog pages (calendars, directories, member lists) are re-checked daily; other pages back off to at most weekly while they keep yielding nothing new. You can force a re-scrape of a whole domain through the API at any time.
Most of the time, very little: an unchanged page is detected by its hash and costs no model call at all, and a changed page is usually read as a diff of the added lines. A domain can carry a monthly spend cap: when it is reached the domain's crawling pauses and a budget.hit event is sent.
Yes. Create a domain, publish a gate (your relevance rules), register source URLs and start a crawl. Autopilot is the same pipeline with the brief, discovery and assessment steps done for you.
