AI crawler log analyzer

Use this AI crawler log analyzer to count visits by bot, path, status and day. Compare source IPs with official ranges locally. Your log is never uploaded.

Read your server log

Nothing is uploaded. Your log stays on your computer.

Plain text, CSV or .gz. Up to 250 MiB, including decompressed content. One format per file.

Sample lines are fictional and only demonstrate the report.

Server logs record requests that reached your server. This free analyzer groups known crawler user agents and shows daily visits, requested paths and response codes. A visit shows a fetch, not a citation or a recommendation. User agents can be faked, so the counts are claims until you verify the source IPs.

How this works

Read your log
Pick a plain text, CSV or gzip file, or paste lines. Apache and Nginx combined logs, Caddy JSON lines, Cloudflare Logpush JSON and CloudFront with its #Fields header are detected automatically. CSV needs user agent, path, status and timestamp columns; their order does not matter.
Count requests
Known crawler tokens are matched as whole user agent tokens. Each operator gets its own totals. Other known crawlers are grouped under Other crawlers; unclassified requests stay separate. Dates are shown in UTC and query strings are removed from paths.
Keep memory bounded
The limit is 250 MiB for the file and for its decompressed content, 256 KiB per record, 100,000 distinct crawler paths, 100,000 distinct error and path pairs across groups, and 3,660 days per group. Parsing stops with an error when a limit is reached. The top 20 paths and top 20 error paths per group are exact within those limits.
Check identity
Download cached official address lists, then compare source IPs locally. Shared ranges support the operator claim only. Missing lists, absent IPs and stale copies remain visible; there is no reverse DNS request.

Questions about crawler logs

No. A log entry records a request. It does not show what an assistant answered, whether it cited the page or whether the visitor saw a recommendation.

Google-Extended is a robots.txt usage control token, not a separate HTTP crawler. Googlebot makes the requests. Applebot-Extended also controls how crawled content is used and makes no separate requests.

The selected period may contain no visits, or your CDN may answer requests before they reach your server. Check edge logs as well as origin logs. An absent user agent header also prevents classification.

Malformed records are counted and excluded from the summary. The detected format applies to the whole file. Analyze logs with different formats separately.

Use user_agent, path, status and timestamp, in any order. Common aliases such as UserAgent, uri, status_code, time and Cloudflare field names are also accepted. Quoted fields, escaped quotes and multiline fields are supported.

Related tools

  • AI crawler check

    Use this AI crawler checker to review robots.txt rules, indexing signals and HTTP responses. Inspect blocked access before changing your site's rules.

  • Crawler IP checker

    Use this crawler IP checker to compare up to 50 addresses with published bot ranges. Paste IPs or a log excerpt, then review the operator matches.

  • AI referral traffic in GA4

    Track AI referral traffic with a GA4 source filter. Choose assistants, test the regex against your source list and copy it into your report.

See whether assistants name your business

Crawler visits show access. The free AI visibility check shows the businesses assistants name in their answers.

Run the free AI visibility check