HomininBot

HomininBot is the web crawler for Hominin search. It fetches public web pages so they can appear in Hominin's search results.

Until 26 September 2026 it was called HominBot. Rules you wrote for that name still apply: it obeys a User-agent: HominBot group as well as its own, and where the two differ it follows the stricter one.

Recognising it

Requests from HomininBot carry a User-Agent like:

HomininBot/1.0 (<crawler name>; +https://hominin.com/bot)

Controlling it with robots.txt

HomininBot follows robots.txt (RFC 9309). It reads the group for User-agent: HomininBot, or User-agent: * when there is none. To keep HomininBot out of your whole site:

User-agent: HomininBot
Disallow: /

A page marked noindex or none, in a <meta name="robots"> tag or an X-Robots-Tag header, is not indexed. Directives addressed to HomininBot or HominBot by name count too.
Crawl-delay is honoured, up to 10 seconds between requests to the same site.
Rules are cached for up to a day, so changes can take that long to be seen.
If robots.txt is missing (a 4xx response), the site is treated as allowing crawling. If it cannot be fetched (a 5xx response or a failed connection), HomininBot does not crawl the site and tries again later.
A redirect to another site is checked against that site's rules.
A 429 or 503 response makes HomininBot leave your site alone for the time given in Retry-After, or a minute when there is none, and never longer than ten minutes.
When it fetches a page again, it sends If-None-Match and If-Modified-Since with the ETag and date your server gave last time, so an unchanged page can answer 304 Not Modified without sending its body.
A page is indexed under its rel="canonical" address when that is on the same site.

Telling it about your pages

HomininBot reads the sitemaps named in your robots.txt (Sitemap: lines), or /sitemap.xml, the first time it reaches your site, and fetches the most recently changed pages first.

It also accepts IndexNow submissions, so you can tell it the moment a page is added or changed:

https://hominin.com/api/indexnow?url=https://example.com/page&key=YOUR_KEY

or POST the usual JSON body (host, key, urlList) to the same address. Your key file must be served at https://example.com/YOUR_KEY.txt. Up to 10,000 URLs a day per site are queued, and only URLs on the site that serves the key.

What it collects

The text and title of a page, the links on it, and the addresses and descriptions of images on it. Pages behind a login are not crawled.

Contact

Questions about HomininBot, or requests about how it crawls your site: hello@hominin.com.