# The Spread the Word crawler

Spread the Word fetches public pages and APIs on behalf of its users, to find conversations relevant to their products and to track replies to what they published.

## How to recognise it

Requests carry the User-Agent `SpreadTheWord/<version> (+https://spread-the-word.steadycircuit.com/bot)`.

## How it behaves

- Reads only public pages and official public APIs. It never signs in to read.
- Respects robots.txt (RFC 9309), cached for up to a day per site.
- Paces requests per host: no more than one request every one or two seconds to any site, and backs off when it receives errors or rate limits.
- Fetches only places a user has approved and threads found there, not whole sites.

## Opting out

Disallow `SpreadTheWord` in your robots.txt and it will stop fetching your site within a day:

`User-agent: SpreadTheWord` / `Disallow: /`

## Contact

Questions or problems: hello@steadycircuit.com.

Updated 2026-10-15. Canonical page: https://spread-the-word.steadycircuit.com/bot