Projects and domains#
A project is a tenant: it holds members, API keys and any number of domains. A domain is one self-contained knowledge graph — its own ontology, sources, gate, page budget, spend cap, entities, relationships and provenance. Nothing crosses domain boundaries; the same organization in two domains is two entities.
Entities, facets and relationships#
Domains default to the POLE model: person object location event. Each kind has a facet — role and aliases; category and identifier; lat, lng and address; starts_at and ends_at. Relationships use 15 canonical predicates with a fixed direction.
Entities are unique per domain by kind and name. Two names for the same official page become one entity with an alias, and curators can rename, merge and delete.
Custom attributes#
Fields the facets lack — contact details, opening hours, price ranges — are custom attributes: up to six per domain, each with a snake_case key, a label and the kinds it applies to. Autopilot creates them when the prompt asks; otherwise set them with PATCH /v1/domains/{id}:
{"attributes": [
{"key": "email", "label": "Email", "kinds": ["object", "location"], "description": "Contact email"},
{"key": "opening_hours", "label": "Opening hours", "kinds": ["location"]}
]}Values are kept only when they appear verbatim on the page the model read, and phone and email keys are shape-checked. A new attribute sends the domain's pages back for a full read on their next re-scrape.
The gate#
The gate (the API calls it the spec) decides what belongs. It has plain-language instructions the extraction agent follows, URL include_patterns and exclude_patterns, allowed_hosts beyond the source hosts, and a language for descriptions. Gates are versioned drafts; exactly one is published. Publishing with ?rejudge=1 gives pages the old gate rejected another look.
Pages and crawl runs#
Every URL Nolvin visits is a page with a status:
| Status | Meaning |
|---|---|
queued | Waiting for a worker. |
extracted | Relevant; entities stored. |
irrelevant | Read and judged outside the gate. |
skipped | Not read: robots.txt, refused content, cancelled run. |
failed | Fetch or extraction failed; after three attempts it becomes a dead letter. |
A crawl run groups the pages of one crawl with counters for pages, tokens, spend and entities added. Its last_mode per page tells you whether a model read it (agentic) or change detection answered (unchanged, deterministic).
Autopilot#
Autopilot builds a domain from a sentence: brief (scope, rules, expected examples, queries, attributes) → discover (validated sources) → crawling → assess (hygiene, precision sweep, geocoding, a judge) → grow (gap sources, frontier plan) — repeated for max_cycles — then steady, with scheduled re-scrapes and re-assessment every two weeks. A failed stage can be retried. See How it works.
Provenance and lifecycle#
Every entity and relationship stores the page it came from and a verbatim snippet. When a page stops mentioning an entity, that page's support is withdrawn; an entity with no support left is removed after 21 days unless it is mentioned again. Curator deletes leave a tombstone, so a re-scrape does not resurrect them.
Freshness and cost#
A scheduler re-checks due pages every six hours: catalogs daily, other pages backing off to weekly (articles to two weeks) while they yield nothing. An unchanged page costs no model call; a changed one is read as a diff. max_spend_usd caps a domain's model spend per calendar month — reaching it pauses the domain and sends budget.hit. GET …/usage reports spend per day.