> ## Documentation Index
> Fetch the complete documentation index at: https://docs.wudlet.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Crawler Analytics Overview
> Gain real-time visibility into how AI systems, search engines, and LLM crawlers read and interact with your website.
## Overview
**Wudlet Crawler Analytics** provides enterprise-grade, server-side visibility into how artificial intelligence systems interact with your digital presence. As AI search engines and conversational assistants become primary discovery channels, understanding and optimizing for AI visibility is crucial for modern digital strategy.
Every AI visit is a potential new customer touchpoint. When users ask ChatGPT, Claude, or Perplexity for product recommendations, vendor comparisons, or technical answers, these systems actively retrieve live web content to synthesize their answers. Wudlet reveals this hidden layer of traffic, giving you the data needed to dominate Generative Engine Optimization (GEO).
Detect AI crawlers and real-time retrieval agents as they access your web pages.
Distinguish between live user-prompt retrieval, search indexing, and LLM training scrapes.
Measure how often major AI platforms fetch your content to answer live user queries.
Identify which pages AI models depend on most and uncover indexing blind spots.
***
## The AI Analytics Challenge
### The Rise of AI Web Interactions
AI systems are fundamentally transforming how information is discovered and consumed online. Conversational search engines like ChatGPT Search, Claude, Google Gemini, and Perplexity don't just index your keywords—they interpret, synthesize, and present your content directly to users in "zero-click" conversational responses.
If your website isn't actively crawled, understood, and retrieved by these AI agents, your brand remains invisible in the fastest-growing search channels.
### Why Traditional Analytics Fall Short
Traditional web analytics platforms (such as Google Analytics, Mixpanel, or Segment) were built for human visitors in standard web browsers. They rely on:
* Executing client-side JavaScript tracking snippets
* Storing client cookies and maintaining browser sessions
* Tracking human mouse movements, clicks, and page flows
AI systems operate completely differently:
* **They bypass JavaScript execution:** AI bots fetch raw HTML and markdown directly without executing client-side scripts.
* **They maintain no session state:** AI crawlers do not store cookies or track user sessions.
* **They access content headlessly:** Requests arrive via specialized server-side crawlers and retrieval agents.
* **They exhibit distinct behavioral patterns:** Crawl patterns vary widely depending on whether the bot is indexing search data, training foundational models, or answering a live user prompt.
Because traditional analytics tools rely on client-side JavaScript, they fail to record AI traffic entirely—recording 0 visits or miscategorizing AI agents as generic server errors or bounce traffic.
***
## The Wudlet Solution
Wudlet Crawler Analytics is engineered from the ground up specifically to monitor AI systems:
### 1. Server-Side & Edge-First Architecture
By integrating directly at your infrastructure level (such as via Cloudflare Workers or Vercel Log Drains), Wudlet captures 100% of incoming requests before caching occurs, ensuring complete visibility into all AI interactions.
### 2. Zero Latency Impact
Wudlet executes telemetry collection completely out-of-band using asynchronous background handlers (such as `event.waitUntil()`). Your origin server and user responses experience **0ms of added latency**.
### 3. Granular Bot Intent Classification
Not all AI bot visits have the same business impact. Wudlet automatically classifies every bot visit into actionable categories:
| Intent | Description | Examples | Business Value |
| :------------- | :-------------------------------------------------------------------------------- | :------------------------------------------------------- | :--------------------------------------------------------------------------------------- |
| **Retrieval** | Live page fetches triggered by an active user asking a prompt in an AI assistant. | `chatgpt-user`, `claude-user`, `perplexity-user` | **High:** Directly indicates potential customers researching your category in real time. |
| **Indexing** | Crawlers scanning and cataloging your site to update AI search indexes. | `oai-searchbot`, `perplexitybot`, `googlebot`, `bingbot` | **Critical:** Ensures your latest content is discoverable in AI search results. |
| **Training** | Crawlers harvesting web content to train future foundational LLMs. | `gptbot`, `claudebot`, `ccbot`, `bytespider` | **Strategic:** Helps determine if your data is being used to train core model weights. |
| **Validation** | Automated checkers validating ad compliance or content policies. | `oai-adsbot` | **Operational:** Maintains compliance with platform standards. |
### 4. AI Answer Volume
The **AI Answer Volume** metric tracks the volume of on-demand retrieval requests coming from top AI operators (OpenAI, Anthropic, Perplexity). This metric serves as a leading indicator of brand relevance: when this number grows, AI models are actively citing and referencing your site to answer real user questions.
### 5. Content Visibility & Top Paths
See exactly which URLs, documentation sections, or blog posts are visited most by AI agents. This helps you:
* Identify high-value content that AI assistants rely on for accurate answers.
* Spot neglected pages or orphaned content that AI bots are failing to discover.
* Optimize site structure and metadata to maximize AI ingestion.
***
## Supported AI Platforms & Crawlers
Wudlet continuously tracks and classifies bots from all major AI platforms and research labs:
| Operator / Platform | Crawlers & Agents | Primary Intents |
| :------------------ | :------------------------------------------------------ | :---------------------------------------- |
| **OpenAI** | `chatgpt-user`, `oai-searchbot`, `gptbot`, `oai-adsbot` | Retrieval, Indexing, Training, Validation |
| **Anthropic** | `claude-user`, `claude-searchbot`, `claudebot` | Retrieval, Indexing, Training |
| **Perplexity** | `perplexity-user`, `perplexitybot` | Retrieval, Indexing |
| **Google** | `googlebot`, `google-extended` | Indexing, Training |
| **Microsoft** | `bingbot` | Indexing |
| **Apple** | `applebot` | Indexing |
| **Meta** | `meta-externalagent` | Training |
| **ByteDance** | `bytespider` | Training |
| **Cohere** | `cohere-ai` | Training |
| **Common Crawl** | `ccbot` | Training |
***
## Implementation Approaches
Wudlet offers non-intrusive integration options designed to fit seamlessly into any technology stack:
### Edge & CDN Integrations (Recommended)
Capturing traffic at the CDN level delivers maximum accuracy before cache hits:
* **Cloudflare:** Automated 1-click Worker deployment through the Wudlet dashboard using the Cloudflare REST API.
* **Vercel:** Native Log Drain integration that streams real-time access logs directly to Wudlet.
* **AWS CloudFront:** Deploy via CloudFormation or Lambda\@Edge for seamless log routing.
* **Google Cloud CDN, Fastly, Netlify & Akamai:** Automated log forwarding and worker scripts.
***
## Enterprise Security & Data Privacy
Wudlet is built to meet strict enterprise compliance standards:
* **Zero PII Requirement:** Wudlet only analyzes crawler metadata (User-Agent, request path, response status, and anonymized IP). No customer personal data is ingested or stored.
* **Ephemeral Credentials:** Third-party API credentials (such as Cloudflare API tokens) are utilized only in-memory during initial deployment and are discarded immediately.
* **Encryption Everywhere:** All telemetry data is encrypted in transit via TLS 1.3 and encrypted at rest using AES-256.
* **Role-Based Access Control (RBAC):** Restrict analytics access by organization and project level.
***
## Getting Started
Ready to see how AI engines are reading your website?
Follow step-by-step guides to connect Cloudflare, Vercel, or custom servers in minutes.
Explore how Crawler Analytics complements Wudlet's AI Search Visibility tracking.