Hominin

HominBot

HominBot is the web crawler for Hominin search. It fetches public web pages so they can appear in Hominin's search results.

Recognising it

Requests from HominBot carry a User-Agent like:

HominBot/1.0 (<crawler name>; +https://hominin.com/bot)

Controlling it with robots.txt

HominBot follows robots.txt (RFC 9309). It reads the group for User-agent: HominBot, or User-agent: * when there is none. To keep HominBot out of your whole site:

User-agent: HominBot
Disallow: /

• Crawl-delay is honoured, up to 10 seconds between requests to the same site.
• Rules are cached for up to a day, so changes can take that long to be seen.
• If robots.txt is missing (a 4xx response), the site is treated as allowing crawling. If it cannot be fetched (a 5xx response or a failed connection), HominBot does not crawl the site and tries again later.
• A redirect to another site is checked against that site's rules.

What it collects

The text and title of a page, the links on it, and the addresses and descriptions of images on it. Pages behind a login are not crawled.

Contact

Questions about HominBot, or requests about how it crawls your site: homininglobal@protonmail.com.