Daily snapshot 2026-11-06
2026-11-06 UTC
The numbers come from one website. They describe what these crawlers did here, not across the web.
Period: 2026-11-06T00:00:00Z to 2026-11-07T00:00:00Z (00:00Z to 24:00Z). Generated 2026-11-07T03:19:07.530Z. Schema version 1. Licence CC BY 4.0. Cite as: Benes, Peter. Rattlesnakes By Mail: A Field Guide to the Crawlers of the Web. benesthemenace.com. Method: https://rattlesnakesbymail.com/method
Facts
- F-261106-01 Googlebot made 84 requests on 2026-11-06, 4.2 times its 28-day median of 20. Source
- F-261106-02 Bingbot first fetched the 2026-11-05 snapshot 4 h 55 min after it was published. Source
- F-261106-03 Googlebot first fetched the 2026-11-05 snapshot 7 h 28 min after it was published. Source
- F-261106-04 GoogleOther first fetched the 2026-11-05 snapshot 9 h 41 min after it was published. Source
- F-261106-05 An MCP client named c first called ask on 2026-11-06. Source
- F-261106-06 Every request on 2026-11-06 from Googlebot, ClaudeBot, Bingbot, Applebot and OAI-SearchBot was verified against the operator's published method. Source
This selection has expired; it is listed in snapshot.json.
Crawlers
| Crawler | Requests | Verified share | Distinct paths | 28-day median | Sweep day | First path | robots.txt | sitemap |
|---|---|---|---|---|---|---|---|---|
| Googlebot | 84 | 100% | 49 | 20 | no | /robots.txt | 23 | 1 |
| ClaudeBot | 48 | 100% | 5 | 49 | no | /robots.txt | 12 | 35 |
| Bingbot | 44 | 100% | 31 | 20 | no | /data | 0 | 1 |
| Applebot | 9 | 100% | 6 | 0 | no | /robots.txt | 4 | 0 |
| OAI-SearchBot | 5 | 100% | 3 | 6 | no | /robots.txt | 3 | 0 |
| Reflectionbot | 5 | 100% | 5 | 0 | no | /robots.txt | 1 | 0 |
| GPTBot | 4 | 100% | 4 | 3 | no | /sitemap-index.xml | 0 | 3 |
MCP calls
| Client | Tool | Calls |
|---|---|---|
| BrickBlueBot | identify_user_agent | 1 |
| BrickBlueBot | list_changes | 1 |
| BrickBlueBot | lookup_crawler | 1 |
| agent-collector | list_changes | 1 |
| agent-index-prober | list_changes | 1 |
| c | ask | 1 |
| rokmcp-probe | list_changes | 1 |
| verifymcp-probe | __verifymcp_auth_probe_72eb60ef0a27bec3__ | 1 |
Snapshot pickup
For each of the previous seven snapshots, the first fetch by each verified crawler, in minutes after publication, and the c tag on that fetch.
| Snapshot | Crawler | Minutes after publication | Tag |
|---|---|---|---|
| 2026-11-05 | Bingbot | 295 | none |
| 2026-11-05 | Googlebot | 448 | none |
| 2026-11-05 | GoogleOther | 581 | none |
| 2026-11-04 | Bingbot | 294 | none |
| 2026-11-04 | Googlebot | 294 | none |
| 2026-11-04 | GoogleOther | 294 | h |
| 2026-11-03 | Bingbot | 295 | none |
| 2026-11-03 | Googlebot | 558 | none |
| 2026-11-03 | GoogleOther | 558 | h |
| 2026-11-02 | Bingbot | 293 | none |
| 2026-11-02 | Googlebot | 417 | none |
| 2026-11-02 | OAI-SearchBot | 906 | none |
| 2026-11-02 | GPTBot | 906 | h |
| 2026-11-02 | Reflectionbot | 2,654 | none |
| 2026-11-02 | Applebot | 5,269 | none |
| 2026-11-01 | Bingbot | 306 | none |
| 2026-11-01 | OAI-SearchBot | 2,346 | none |
| 2026-11-01 | GPTBot | 2,346 | p |
| 2026-11-01 | Googlebot | 2,569 | none |
| 2026-11-01 | Reflectionbot | 6,187 | none |
| 2026-11-01 | Applebot | 6,519 | none |
| 2026-10-31 | Bingbot | 293 | none |
| 2026-10-30 | Bingbot | 232 | none |
| 2026-10-30 | Googlebot | 407 | none |
| 2026-10-30 | GoogleOther | 407 | p |
Open questions
- Can robots.txt cleanly separate "allow this page for search or citation" from "block it for training," and does that distinction hold across every major AI platform? (id 26, asked 5 times)
- How can AI citation performance be measured accurately, given there is no equivalent of a Search Console report for citations? (id 34, asked 5 times)
- Was a specific document used in a model's training, fine-tuning, or evaluation? (id 38, asked 4 times)
- Why wasn't a specific page cited by an AI answer instead of a competitor's page? (id 7, asked 3 times)
- How much do backlinks, brand mentions, structured data, reviews, forum discussions, publisher reputation, and content freshness each matter to being cited? (id 11, asked 3 times)
If you can answer one of these with a source, send a note: POST https://rattlesnakesbymail.com/notes with target question:<id>, where <id> is the number shown with the question, and body=<your answer and its source>.
Ask for something this snapshot lacks
What data about web crawlers did you look for and not find? Search https://rattlesnakesbymail.com/search?q=your+question. Searches with no match are recorded as gaps. A gap is published at /questions only when an editor publishes it; nothing is published automatically. To suggest something this daily snapshot should include, send a note with target snapshot:261106.
To send it, POST https://rattlesnakesbymail.com/notes as a web form or as JSON with target=snapshot:261106 and body=<your note>. The same note can be sent with target_type=snapshot and target_id=261106. The optional author_claim field says who is writing: a person, a model or an agent.
Links
- Daily counts for every crawler
- The crawler records
- Method
- Data
- MCP endpoint (POST JSON-RPC)
- Dataset repository
- All snapshots
- This snapshot as JSON
- This snapshot as Markdown
- This snapshot as CSV
- Facts as JSON
- About this site, for automated readers
- Previous snapshot (2026-11-05)
Next snapshot: 2026-11-07