ScoutyBot

Identification

First-party website requests, including robots.txt and redirects, use this user agent:

ScoutyBot/1.0 (+https://scouty.to/bot)

A user agent alone does not prove a request came from Scouty. Third-party search and data providers operate their own retrieval infrastructure and identities.

What we retrieve

ScoutyBot reads a business homepage and up to three relevant linked pages, such as about, services or pricing pages. Requests have time, response-size and redirect limits. It does not sign in to websites. When enabled, Scouty may use Firecrawl to retrieve public pages that could not be read directly, including pages with browser verification or JavaScript rendering. Research results may be cached and reused for campaign and outreach preparation.

Control access

ScoutyBot checks robots.txt before retrieving pages, including redirected destinations. To block it, publish:

User-agent: ScoutyBot
Disallow: /

We respect matching Allow and Disallow rules. When a matching policy requests a positive Crawl-delay, ScoutyBot conservatively declines retrieval. Direct retrieval stops when a robots policy is unavailable, rate limited or cannot be verified; an enabled Firecrawl fallback may still attempt retrieval in that case. HTTP 404 and 410 mean no published policy. Known matching restrictions also prevent fallback. Firecrawl operates its own requests and redirects; Scouty checks requested and returned URLs but cannot inspect every intermediate provider redirect.

Contact

For bot traffic questions or access concerns, contact [email protected]. Include the affected domain and request timestamps, with any credentials or personal information removed.