Overview
Agent Traffic is Scrunch’s bot analytics tool that monitors access from AI bots like ChatGPT, Perplexity, Gemini, and Grok. This guide shows you how to stream traffic logs from Fastly to Scrunch using HTTPS log streaming.
Fastly HTTPS log streaming sends real-time access logs to Scrunch’s webhook endpoint, allowing near-instant visibility into bot traffic on your site.
Note: This is a pilot feature and you need to ask Scrunch customer service to enable and onboard you.
What You’ll See
Once integrated, the Agent Traffic dashboard will show:
Total bot traffic metrics - Overall volume and trends
Traffic distribution by bot type - Breakdown between Retrieval bots (ChatGPT), Indexer bots (Google), and Training bots (GPTBot)
Top bot agents - Which specific bots are accessing your site most frequently
Top content pages accessed - Which pages bots are visiting
Traffic timeline - Historical view of bot activity
Status code distribution - Success rates (200s) vs errors (404s, etc.)
Method breakdown - GET, POST, and other HTTP methods used by bots
Date filtering options - View data for the last 24 hours, 7 days, or 30 days
Adding Your Website
Before configuring Fastly, add your domain in Scrunch to get credentials:
Log into your Scrunch account at app.scrunchai.com
Click Agent Traffic in the main menu
Click Connect Site or Add Domain
Enter your domain name (e.g.,
example.com)Select Fastly as the platform
Click Create
After creating the site, you’ll see two credentials on the site detail page:
Webhook URL - Format:
https://webhooks.scrunchai.com/v1/sites/[SITE_ID]/platforms/fastly/web-trafficAuthentication Token - A long string starting with
eyJ...
Keep these credentials handy for the next section.
Integrating Fastly HTTPS Log Streaming
Step 1: Open Fastly Control Panel
Log into the Fastly Control Panel
Navigate to Services and select your CDN which contains the domain you added above to ScrunchAI
Click Service configuration → Clone to edit to create a draft (new version, inactive)
Step 2: Create HTTPS Logging Endpoint
In the left sidebar, click Logging
Scroll to the HTTPS section and click Create endpoint
Step 3: Configure Basic Settings
Enter the following in the endpoint form:
Name:
Scrunch Agent TrafficPlacement: Format Version Default
Log Format: Check below for the content to paste there
URL: Paste your Webhook URL from Scrunch
Maximum logs: 10000
Maximum bytes: 10485760
Processing region: US
Content type: application/json
Custom header name:
X-Api-KeyCustom header value:
[YOUR_TOKEN](replace[YOUR_TOKEN]with your authentication token from Scrunch)Important: Do NOT include
Bearerbefore the Token.
Method: POST
JSON log entry format: Array of Json
Select a log line format: Blank
Compression: None ← Make sure to select this, the default value will not work unless it’s explicitly None (Pure JSON)
Using your own certificate authority (CA): Leave all fields under this empty
Press Save after all of the above is entered. You will go back to the Version page.
Click on “Activate” so it starts applying
You are activating a new Version of Configuration which now includes this new Logging Endpoint
Log format:
{ "timestamp": "%{begin:%Y-%m-%dT%H:%M:%SZ}t", "host": "%{req.http.Host}V", "method": "%m", "path": "%{json.escape(req.url.path)}V", "status": %{resp.status}V, "user_agent": "%{json.escape(req.http.User-Agent)}V", "client_ip": "%h", "pop": "%{server.datacenter}V", "service_id": "%{req.service_id}V" }Step 4: Verify Traffic
After activation, generate some test traffic:
Visit your website:
https://yourdomain.comMake sure you are reaching your website via the domain you configured in Fastly!
Navigate to a few different pages
Wait 30-60 seconds for logs to appear
Check your Scrunch Agent Traffic dashboard , you should see the traffic within 1-2 minutes.
Your website should be active and you should be seeing the bot traffic for that domain now!
Troubleshooting and Tips
No traffic appearing after 5 minutes?
Test the endpoint: In Fastly Control Panel → Logging → Find your Scrunch endpoint → Click Test. Fastly will send a test request.
Check for errors: Look for 401, 403, or 404 errors in the test result
Verify Authorization header: Make sure you included
Bearerbefore the tokenCheck service activation: Ensure you clicked “Activate” (not just “Save”)
Verify domain: If using domain filtering, make sure the filter condition is correct
Common issues:
401 Unauthorized: Authorization header is missing
Bearerprefix or token is incorrect404 Not Found: Webhook URL is incorrect (check the site ID in the URL)
No test traffic showing: Make sure you’re visiting the correct domain that matches your filter condition
Tips for multiple domains:
If you have multiple domains on one Fastly service, create a separate site in Scrunch for each domain, then:
Create separate HTTPS logging endpoints for each domain
Name them distinctly:
Scrunch - example.com,Scrunch - shop.example.comUse domain filtering to ensure each endpoint only sends logs for its specific domain
Data appears delayed:
Fastly batches logs every 10 seconds or when reaching 10,000 entries / 10 MB. During low traffic periods, logs may take up to 10 seconds to appear. This is normal.
Optimizing data quality:
Fastly HTTPS logging is free but counts toward bandwidth usage (logs are gzipped, so impact is minimal)
If you see excessive unmatched domains in Scrunch, add domain filtering to reduce noise
Consider filtering out static assets (CSS, JS, images) using response conditions if needed
For additional help, contact [email protected] or use the chat widget in your Scrunch dashboard.