sign in →

for site owners / crawler

NolvinBot

If you found this page in your server logs: NolvinBot reads public pages to keep knowledge graphs about events, organizations and places up to date. Here is how it behaves and how to control it.

crawleruser-agent: nolvinbot

How it identifies itself#

Every request carries this user agent:

NolvinBot/0.1 (+https://nolvin.com/bot)

Geocoding lookups against OpenStreetMap's Nominatim use a similar NolvinBot user agent.

robots.txt#

NolvinBot fetches /robots.txt before crawling a host and caches it for 24 hours. It follows the group for NolvinBot if there is one, otherwise the * group, with longest-match Allow and Disallow rules. A disallowed page is not fetched.

To keep NolvinBot out entirely:

User-agent: NolvinBot
Disallow: /

To keep it out of part of a site:

User-agent: NolvinBot
Disallow: /members/
Disallow: /search

Request rate#

Requests to the same host are spaced at least 1.5 seconds apart across all of Nolvin's workers. A Crawl-delay in robots.txt is honored up to 30 seconds:

User-agent: NolvinBot
Crawl-delay: 10

What it reads, and how often#

  • Only public HTML pages, reached from sources a knowledge domain registered or from links on those pages. It sends plain GET requests — no logins, no form submissions — and skips login, social-media and ticket-shop links.
  • Pages are re-checked on a schedule — calendars and directories about daily, other pages backing off to weekly or less while they do not change.
  • Content is used to extract facts — names, dates, places, relationships — each stored with a link back to your page and a short quote. Pages are not republished.

Contact#

Seeing unexpected traffic, or want a site removed from all graphs? Blocking in robots.txt takes effect within 24 hours. For anything else, reach the team through the feedback button in the console.