Googlebot crawler guide
The Googlebot crawler guide,
as documented
What Google documents, in its own words. Verified from Google's page on 2026-10-09.
Google's Googlebot page describes its two crawler types, how it accesses a site and how to block or verify it. This page lists those statements with the exact wording from Google.
What Google says
| Topic | What Google says | Source |
|---|---|---|
| Two crawler types | “Googlebot is the generic name for two types of web crawlers used by Google Search:”Google names Googlebot Smartphone and Googlebot Desktop. | Google docs Verified 2026-10-09 |
| One robots.txt token | “both crawler types obey the same product token (user agent token) in robots.txt, and so you cannot selectively target either Googlebot Smartphone or Googlebot Desktop using robots.txt.” | Google docs Verified 2026-10-09 |
| Mobile first | “For most sites Google Search primarily indexes the mobile version of the content.” | Google docs Verified 2026-10-09 |
| Crawl rate | “For most sites, Googlebot shouldn't access your site more than once every few seconds on average.” | Google docs Verified 2026-10-09 |
| Slowing it down | “If your site is having trouble keeping up with Google's crawling requests, you can reduce the crawl rate.” | Google docs Verified 2026-10-09 |
| File size limit | “When crawling for Google Search, Googlebot crawls the first 2MB of a supported file type, and the first 64MB of a PDF file.” | Google docs Verified 2026-10-09 |
| Past the cutoff | “Once the cutoff limit is reached, Googlebot stops the fetch and only sends the already downloaded part of the file for indexing consideration.” | Google docs Verified 2026-10-09 |
| Uncompressed data | “The file size limit is applied on the uncompressed data.” | Google docs Verified 2026-10-09 |
| Other crawlers | “Other Google crawlers, for example Googlebot Video and Googlebot Image, may have different limits.” | Google docs Verified 2026-10-09 |
| Timezone | “When crawling from IP addresses in the US, the timezone of Googlebot is Pacific Time.” | Google docs Verified 2026-10-09 |
| Secret sites | “It's almost impossible to keep a site secret by not publishing links to it.” | Google docs Verified 2026-10-09 |
| Crawling versus indexing | “there's a difference between crawling and indexing; blocking Googlebot from crawling a page doesn't prevent the URL of the page from appearing in search results” | Google docs Verified 2026-10-09 |
| Block access fully | “Prevent a page from being accessible at all by both crawlers or users? Use another method, such as password protection.” | Google docs Verified 2026-10-09 |
| Spoofed user agent | “the HTTP user-agent request header used by Googlebot is often spoofed by other crawlers.” | Google docs Verified 2026-10-09 |
| Verifying | “The best way to verify that a request actually comes from Googlebot is to use a reverse DNS lookup on the source IP of the request, or to match the source IP against the Googlebot IP ranges.” | Google docs Verified 2026-10-09 |
How to apply it
A checklist built only from the guidance in Google's page.
- Use a robots.txt file to stop Googlebot crawling a page, and noindex to stop indexing.
- Do not expect to target Googlebot Smartphone or Desktop separately in robots.txt.
- Know the limits: the first 2MB of a supported file and the first 64MB of a PDF, with each CSS or JavaScript resource fetched separately under the same limit.
- If crawling strains your server, reduce the crawl rate.
- Verify suspicious requests with a reverse DNS lookup or the Googlebot IP ranges, not the user-agent string.
Questions
Can I block only Googlebot Smartphone?
Google says both types obey the same robots.txt product token, so you cannot target one without the other.
How much of a page does Googlebot crawl?
Google says the first 2MB of a supported file type and the first 64MB of a PDF file, applied to uncompressed data.
Does blocking Googlebot remove a URL from results?
Google says blocking crawling doesn't prevent the URL from appearing in search results. It points to noindex to stop indexing.
How do I know a request is really Googlebot?
Google says the user-agent header is often spoofed and recommends a reverse DNS lookup or matching the Googlebot IP ranges.
How current is this page?
Every quote was read from Google's page on 2026-10-09. The page showed "Last updated 2026-02-03 UTC". Google can change its documentation, so treat the linked page as the final word.
Sources
- Googlebot (Google Search Central), read on 2026-10-09
Let Balzac write the articles.
Type your website. Balzac finds the searches you can win, writes articles built to rank on Google and get cited by ChatGPT, and publishes them for you.
3 free articles · No card needed