Overview
The Agent Traffic tool in Scrunch lets you monitor how much access your site is getting from LLM bots—including ChatGPT, Perplexity, Gemini, Grok, and others. Akamai Data Stream 2 broadcasts your traffic logs directly to Scrunch.
If your website is proxied by Akamai, then Data Stream 2 is the perfect tool to broadcast traffic and get metrics from LLM bots in real time.
What You’ll See
Once your Akamai Data Stream 2 is connected, the Agent Traffic dashboard will show:
Total Bot Traffic in the last period
Bot traffic over time
Traffic distribution between Retrieval, Indexer, and Training LLM Bots
Comparison between the current period and the last period (%)
Top bot agents and when they were last seen
Top content pages accessed by LLM bots
Recent bot requests
A date filter to see data from the last 24 hours, last 7 days, or last 30 days
Scrunch AI's Agent Traffic feature allows customers to granularly track
which AI platforms are consuming their content (and for what purpose) to enable better understanding of how their content:
will be surfaced in AI platforms like ChatGPT
drives AI responses to relevant questions
and ultimately how it influences AI to describe and recommend their brand,
products and services and click through to their site(s).
Adding Your Website
Open the Scrunch app.
Navigate to the Agent Traffic menu.
You’ll see the list of websites already connected to Agent Traffic.
Click + Connect Site at the top.
Select Akamai as your CDN platform.
A dedicated instructions page will appear, showing your Webhook URL and API Key.
Your page will look like this one:
ℹ️ Each site has its own endpoint and key. Don’t reuse them across different sites or integrations.
Integrating Akamai Data Stream 2
Step 1: Configure DataStream 2
Log in to your Akamai Control Center with DataStream access.
Go to Common Services → DataStream and create a new stream.
Provide a Stream Name and continue.
Requirements: Akamai must be serving as a reverse proxy in front of your site, and your account must include DataStream 2.
Step 2: Select Data Parameters
In the Data sets tab, select the fields below. They're grouped and named as they appear in Akamai's field picker.
Required fields
Log information
Request time
Message exchange data
User-Agent
Protocol type
Request host
Request path
Request method
HTTP status codes
Network performance data
Request end time
Turnaround time
Transfer time
DNS lookup time (if your stream's log format offers it)
⚠️ If any required field is missing, Scrunch rejects every log line and your site stays "Pending."
ℹ️ User-Agent also needs a property change. In Property Manager, add the Log Request Details behavior to your rule (for example, the Default Rule), turn on Log User-Agent Header, then activate the property. Without it, the User-Agent field arrives empty and Scrunch can't identify which AI agents are visiting your site.
Recommended (optional) fields
Message exchange data
Query string
Client IP
Step 3: Configure Destination (Webhook)
In the Destination tab, choose Custom HTTPS Endpoint.
Set Log Format to
JSON.Enter the Webhook URL and X-Api-Key header value from your Scrunch app.
Example:
Webhook URL: https://webhooks.scrunchai.com/v1/sites/<SITE_ID>/platforms/akamai/web-traffic
Header: X-Api-Key = <YOUR_API_KEY>
ℹ️ Akamai does not allow Authorization headers for DataStream, so always use X-Api-Key.
Step 4: Review & Activate Stream
On the Summary step, review your configuration: URL, header, dataset fields, and frequency.
Save and activate the stream.
Akamai will send a one-time verification request to Scrunch.
Step 5: Enable DataStream Behavior on Property
Open Property Manager for your site.
Add the DataStream behavior (v2).
Save and activate your property version.
Step 6: Verify Traffic
Wait up to 5 minutes for your site to show as “Active” in Scrunch.
If you don’t see traffic, test with:
curl -v -H "User-Agent: ScrunchAI-Testbot" https://yourdomain.com
This will send a sample request to confirm logs are flowing.
👉 Once configured, your site will continuously stream traffic logs to Scrunch, giving you real-time visibility into how LLM bots interact with your content.
Troubleshooting and Tips
Don’t see any traffic after integrating?
Ensure the Webhook URL and X-Api-Key match what’s in your Scrunch app.
Verify the DataStream behavior is active in Akamai Property Manager.
Double-check the stream is in Active state in the Akamai console.
Confirm you included all required fields (see Step 2).
Wait 5–10 minutes after activation.
Tips for better results:
Exclude paths for static assets (CSS, JS, images) if you want cleaner data.
Include paths that serve PDFs—AI bots often request them.
Keep stream frequency at 1–5 minutes for near real-time monitoring.
If you manage multiple sites, repeat the process for each site in Scrunch.