To get the most accurate data from Scrunch, you'll need to allowlist our crawlers in your web infrastructure. This lets us access and analyze your pages without getting blocked by firewalls, WAFs, or bot-detection rules.
Once you've completed the setup below, let us know so we can re-enable crawling for your site or kick off a new crawl.
What to allowlist
You'll need to allowlist both our IP addresses and user agents.
IP addresses
Allow traffic from these IPs in your firewall, CDN, or WAF rules:
IP Addresses:
34.42.160.208, 34.133.153.129, 35.223.169.148
User agents
Our crawlers identify themselves with the following user-agent strings. Allowlist these in any bot-detection or traffic-filtering rules you have in place.
Scrunchbot (primary crawler)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Scrunchbot/1.0; +https://scrunchai.com/bots)
ScrunchAI Testbot (used for validation checks)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ScrunchAI-testbot/1.0; +https://scrunchai.com/bots)
Additional user agents (used for specific analysis scenarios)
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36
Where to apply these rules
This depends on your setup, but common places include:
CDN (Cloudflare, Akamai, Fastly, AWS CloudFront) — add IP allowlist rules and bypass bot detection for our user agents
WAF (Web Application Firewall) — create allow rules for our IPs and user agents
Firewall / security groups — allow inbound HTTP/HTTPS from our IPs
robots.txt — make sure
ScrunchbotandScrunchAI-testbotare not disallowed. Example:
User-agent: Scrunchbot Allow: / User-agent: ScrunchAI-testbot Allow: /
If you use rate limiting, consider raising the threshold for our IPs to avoid throttling during crawls.
After you've completed setup
Once your allowlisting is in place, reach out to us to confirm. We need to know you've completed the setup before we can start re-crawling your site. You can let us know via:
Your Scrunch account manager
Email at [email protected]
The in-app chat
We'll verify the configuration on our end and enable crawling for your domain.
Troubleshooting
Crawls are returning errors or incomplete data Double-check that all three IPs and all user agents are allowlisted. Some WAFs require both IP and user-agent rules to pass.
Pages are loading but content looks different Some CDNs serve different content to bot user agents. Make sure our user agents aren't getting a stripped-down or cached version of your pages.
Not sure where to add the rules? Reach out to us and we can help figure out the right configuration for your setup.