Day to day
AI logs: connecting your request logs
Which AI crawlers read which pages, how fast your site answered, and what status they got. Crawlers never run JavaScript, so the tracking snippet can’t see them — this screen reads your site’s own request logs instead. Setup is one URL to paste, or one file to add.
By Garvit Sharma, Co-founder
AI logs (Analyst lens)
What it measures
| Crawler | Role | What it means for you |
|---|---|---|
OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot | Search | Blocking one of these means that assistant can’t cite you today. |
ChatGPT-User, Claude-User, Perplexity-User | User fetch | A person asked the assistant to read this page. |
GPTBot, ClaudeBot, Meta-ExternalAgent, Amazonbot, CCBot | Training | Bulk collection. It pays off over model generations, not today. |
Only AI crawlers are kept. Ordinary visitors, Googlebot and uptime monitors are dropped on arrival, so you can send every request without filtering anything first.
How it connects
Open AI logs and find Connect your logs. Pick where your site is hosted, and the screen shows either a URL to paste into that platform’s log settings or a file to add to your site — with your site’s ingest URL already inside. The URL contains a secret key: it lets a shipper post log lines for your site and nothing else, and it is not the public key in the tracking snippet.
| Hosted on | What you do | Needs |
|---|---|---|
| Vercel | Paste the URL as a drain (Settings → Drains), NDJSON, with the page-request sources enabled | Pro or Enterprise. On Hobby, use the Next.js file instead |
| Cloudflare | Add the Worker the screen gives you, routed on your whole site — or, with a Cloudflare connection, switch on Send AI crawlers to AI logs | Any plan (100,000 requests a day on Free). Enterprise can paste the URL into Logpush instead |
| Netlify | Paste the URL as a General HTTP log drain, Traffic logs, PII exclusion off | Enterprise |
| Next.js on Render, Fly, Docker or Vercel Hobby | Add proxy.ts (middleware.ts before Next 16) and deploy | Any host |
| WordPress | Save the plugin file under wp-content/mu-plugins/ | File access. A full-page cache hides the hits it serves itself |
| nginx or Apache on your own server | Run Vector with the config the screen gives you | Any |
| Wix, Shopify, Squarespace, Webflow, Framer | Not available: the platform exposes no server logs | AI traffic still works there |
| Anything else | POST NDJSON or JSON to the URL from wherever you see requests | Any |
Copy instructions for your developer hands over the steps, the URL and the file as one message.
Check the connection
Press Check connection. Vidrys fetches your site as VidrysCheck/<code> and waits up to 90 seconds for that request to come back through your setup. It works before any AI crawler has visited, so you never wait days to learn whether the pipe is connected.
| The screen says | It means |
|---|---|
Receiving logs | Batches are arriving. The line shows when the last one landed, how many lines were checked, and how many AI crawler hits were found in them. |
Connected end to end | Our check request came back through your setup. |
Logs are arriving, but our check request didn’t come through | Something posts to the URL, but page requests are not part of it. On Vercel and Netlify, enable the page-request log types; for a file, make sure it runs on every path. |
Nothing arrived in 90 seconds | The URL isn’t saved, the platform’s own test failed, or the file isn’t deployed. Fix that, then check again. |
Not connected | No batch has arrived in the last seven days. |
Sending logs yourself
Any process that sees requests can POST to the ingest URL. NDJSON (one JSON object per line), a JSON array, or {"entries": [...]} all work, gzipped or plain, up to 10 MB a request. Vercel, Netlify and Cloudflare Logpush payloads are read as they come, and so is a raw nginx or Apache combined line.
POST <your ingest URL>
Content-Type: application/x-ndjson
{"timestamp":"2026-09-19T10:04:11Z","ip":"20.171.206.14","method":"GET","path":"/pricing","status":200,"origin_ms":412,"user_agent":"Mozilla/5.0 (compatible; GPTBot/1.0)"}| Field | Notes |
|---|---|
user_agent | The only required field — it is how a crawler is recognised. |
timestamp | ISO 8601, or a unix epoch in seconds, milliseconds or nanoseconds. Missing: the time it arrived. |
path | The page. Query strings are dropped before anything is stored. |
status, origin_ms, ip, country, method | Optional. Response times appear only when the shipper sends them: on nginx add $request_time to log_format, on Apache %D. |
The reply carries accepted (AI crawler lines stored), skipped and lines. A wrong key answers 401, so a platform’s own test fails loudly at setup instead of silently for ever. A body over the size limit answers 413.
What is kept
One row per AI crawler request: crawler, role, time, method, path, status, response time, the crawler’s IP and country, and its user agent. The IP is kept because it is the only way to tell a real GPTBot from something merely claiming to be one. Nothing about ordinary visitors is stored.