Skip to content
PromotEdge

Home World of Marketing What Google Says About Website

Insights

What Google Says About Website Crawling in 2026

By , SEO executive with Shivanand MishraUpdated 16 Oct 20264 min read

Since 201565+ in-house team1 office in Kolkata · 5 team bases
A marketer in a red bow tie pointing a library visitor down an aisle with a small arrow sign in a Delhi library.
The short answer

In short

How does Googlebot crawl a website? If Google cannot fetch a page properly, no amount of good writing will get it ranked. PromotEdge checks how Google crawls a site and fixes what blocks or slows it before touching the content. Do you picture Googlebot as one single spider that reads every byte of your pages? Time for an update. Google's Gary Illyes wrote a post titled "Inside Googlebot: demystifying crawling, fetching, and the bytes we process." In it, he explains how the search giant's crawling ecosystem actually works in 2026.

Need this done for your business? A strategist can call you back in one working day.

Ask for a call back

What he says has direct effects on your SEO (search engine optimisation). This blog goes through the key points. It explains what matters for your website's visibility, and if you want help to fix Google crawl limits and Googlebot crawl problems on your own site, see our technical SEO services in India. For a short definition, see technical SEO in our glossary.

How does Googlebot crawl a website? A group of crawlers

The word "Googlebot" makes you picture a single bot visiting your site. That picture is outdated. Google runs many different crawlers for different purposes, from indexing HTML pages to fetching images and videos.

Each crawler follows its own rules, including its own byte-fetch limits. How does Googlebot crawl a website? Googlebot is not one bot. Google runs many crawlers for different purposes, and they read only up to a set number of bytes from each page, so keep pages lean and put the important content early. (PromotEdge, summing up Google's post covered on this page, 6 November 2026.) Say you block or allow "Googlebot" in your robots.txt (the file that tells bots where they may go). You must know which bots matter for your content. If you do not, you can leave gaps in what gets indexed.

Googlebot Ecosystem With Multiple Crawlers

The byte limits Google crawlers work under

Google also shares the size limits Googlebot applies when fetching files:

Crawler / Content TypeFetch Limit
Standard Googlebot (HTML pages)2 MB
PDF files64 MB
Image & video crawlersVariable, depends on product
All other crawlers (default)15 MB

That 2 MB limit for HTML includes the HTTP response headers. So your actual HTML gets slightly less than 2 MB.

NOTE: For most websites the 2 MB limit is not an issue. For large e-commerce pages, JavaScript-heavy single-page apps or pages with huge inline data, it can be.

What happens when your page goes over the limit?

Google does not reject oversized pages outright. The process has a few steps. Knowing them explains why some content gets indexed while other parts do not:

Googlebot 2MB Limit And Partial Crawling Explained

Step 1: Googlebot fetches only part of the page. If your HTML file is larger than 2 MB, Googlebot stops the fetch at the 2 MB cutoff and works with what it has.

Step 2: Google treats the cutoff as the whole file. The downloaded part (bytes 1 up to the 2 MB mark) goes to Google's indexing systems and Web Rendering Service (WRS), and they treat it as if it were the complete file.

Step 3: Anything beyond 2 MB is lost. Content past the threshold is never fetched, rendered or indexed. As far as Google is concerned, it does not exist.

Step 4: External files are counted separately. WRS fetches every resource the HTML refers to, each with its own per-URL byte counter, so these files do not count against the parent page's 2 MB.

How Google's Web Rendering Service (WRS) works

Once Googlebot hands over the fetched bytes, the Web Rendering Service (WRS) takes over. It works like a modern browser: it runs JavaScript, processes CSS and handles XHR requests (data requests a page makes in the background). That helps it understand the final look and text of a page.

GOOGLE’S OFFICIAL EXPLANATION : The WRS processes JavaScript and executes client-side code similar to a modern browser to understand the final visual and textual state of the page. Rendering pulls in and executes JavaScript and CSS files, and processes XHR requests to better understand the page’s textual content and structure.
Google WRS Rendering Process Explained

During rendering, WRS does not request images or videos. It asks only for resources that affect text, and for each external resource it pulls in, the same 2 MB per-URL limit applies separately.

Three practices to act on today

Three HTML Best Practices For Performance

Keep your HTML lean

Move heavy CSS and JavaScript into external files. The main HTML document has the 2 MB cap, but each external stylesheet and script file gets its own separate 2 MB. Why bother? Large blocks of code inside your HTML can push your important metadata past the crawl cutoff.

Order matters in your HTML

Put your most important SEO elements as high in the document as possible. That means <title> tags, meta descriptions, canonical links, Open Graph tags and key structured data. Why high? Elements pushed deep into a large HTML file may sit beyond the 2 MB cutoff and never be seen by Google.

Monitor your server response times

If your server is slow to deliver bytes, Googlebot's fetchers automatically slow their crawl rate. Slower servers mean fewer pages crawled per day. It's worse on large sites. So check your server logs regularly for crawl activity and response time.

What this means for your SEO strategy in 2026

SEO Strategy In 2026 Focused On Technical Efficiency

The disclosures from Gary Illyes remind us that technical SEO is not set-and-forget. Google's crawling system is advanced, but it works within real limits such as bandwidth, server load and byte budgets. Sites that get the most complete crawl are the ones that make Googlebot's job easy.

For most well-maintained websites, the 2 MB HTML limit is unlikely to be a problem. An audit is still worth doing for fast-growing e-commerce platforms and news sites with complex templates. It is also worth doing for any site where developers have used the HTML as a dumping ground for inline scripts and styles.

If you want someone to run that audit for you, see our SEO services.

At PromotEdge, we suggest treating the 2 MB limit as a useful push towards leaner, better-structured HTML. Lean HTML also loads faster for your users. Good technical SEO and good user experience point the same way. Simple.

Common questions

Short answers. For anything else, call or WhatsApp us.

  • No. Google runs many different crawlers for different purposes, from indexing HTML pages to fetching images and videos, and each follows its own rules, including its own byte-fetch limits. Do you block or allow Googlebot in robots.txt without knowing which bots matter for your content? Then you can leave gaps in what gets indexed.

  • Standard Googlebot fetches up to 2 MB for HTML pages, and that includes the HTTP response headers. PDF files have a 64 MB limit, and other crawlers default to 15 MB. Image and video crawlers vary by product. So the space left for your actual HTML is slightly less than 2 MB.

  • Google does not reject the page. Googlebot stops the fetch at the 2 MB cutoff and treats the downloaded part as the whole file. Anything beyond that is never fetched, rendered or indexed. As far as Google is concerned, it does not exist. External files are different: they are counted separately, with their own limit.

  • For most well-maintained websites it is not an issue. It can matter for large e-commerce pages, JavaScript-heavy single-page apps and news sites with complex templates, and it can also matter for any site where developers have used the HTML as a dumping ground for inline scripts and styles. In those cases an audit is worth doing.

  • Keep your HTML lean by moving heavy CSS and JavaScript into external files, then put important SEO elements high in the document, such as the title tag, meta description, canonical links and key structured data. Watch your server response times too, because slow servers make Googlebot crawl fewer pages.

What people say

What clients say about PromotEdge

Brands we have worked with

  • Sai Angan
  • Rupa Peekaboo
  • Tea Junction
  • Zandu Vigorex
  • Fox & Mandal
  • Veedol
  • Tata Efee
  • Avadh Sugar & Energy
  • Taj Cement
  • Weichai
  • Rerasathi
  • Honest AI
  • Walton
  • Supertron
  • Balrampur Foundation
  • Fine Organics
  • Kopran
  • Bengal Aerotropolis Projects Limited
  • FIVB
  • Hanumanta Foods
  • Linc
  • Nitya Gold & Diamonds
  • Novozymes
  • PSPL
Fine Organics
“PromotEdge and their team did an excellent job with our website. They engaged with every team member, understanding their needs with great patience before collating them into a user requirement and a design spec. The result was both aesthetically pleasing and usable. PromotEdge continues to support us and our ever-growing and very demanding wish list. We hope to grow this relationship beyond website design and into content development. I am confident that PromotEdge's team will continue to guide us in delivering product content to our global customers, making the website an important tool for product awareness and lead generation.”
Bimal ShahBimal ShahExecutive Director at Fine Organic Industries LimitedManufacturing
Rishta Foods
“PromotEdge is one of the most hard working creative agency that LISTENS ! Do I need to say anything more ?!!!!Saurav is one of the most open person I have come across who is ready to speak abt the elephant in the room , that no body wants to talk about !”
Harsh GuptaHarsh GuptaManaging Director at Rishta FoodsFMCG
iLead
“I am delighted with the successful launch of our new website, thanks to the exceptional support from PromotEdge. I extend my gratitude to Benjamin and the team for their dedication and hard work, which has resulted in a new, fresh-looking site that perfectly meets our expectations.”
Mom MitraMom MitraMarketing Head at IleadEducation
ABP Education
“Working with PromotEdge has been an enriching experience for ABP at many different levels. Your understanding of the digital space, customized brand solutions and prompt servicing have helped us achieve our brand goals with so much ease.I would like to especially thank the creative development team, the digital strategy team and above all the servicing team with whom I have worked in close association with, for their all-round support. I wish you all the best in your upcoming projects and I look forward to collaborating with you in future as well.”
Satabdi SahuSatabdi SahuLead - Brand and Content at ABP EducationEducation
Brahmaputra Volleyball League
“I really want to thank Sayantan and his team from PromotEdge because what I find good in them is that, the team.When the team is wonderful, the outcome will definitely be wonderful. I have seen the first cut of the reel which they shot the first day, and they gave us on the morning of the second day.The entire first day they worked so hard and over the night they have edited and on the morning they shared it with us.This is just fabulous and the commitment they have shown. The FIVB team has said that they are very happy with team PromotEdge.And when I showed this clip to my friends, those who are from this industry, they (also said that they) have done a fabulous work.So happy and compliments to the team -- Sayantan and PromotEdge team.Thank you.”
Abhijit BhattacharyaAbhijit BhattacharyaFormer Captain Indian VolleyBall Team; and Founder of Brahmaputra Volleyball LeagueSports
FIVB
“Hi, my name is Hitesh Malhotra. I'm working in FIVB. We are working with PromoteEdge (with Team Sayantan).I would like to give you a brief detail. So when we started this project, we had a good discussion with them. And personally, I feel PromotEdge is kind of a one stop solution for us. So basically, they provided us the whole storyboarding regarding the digital assets and how we can use those with our ambassadors programme.They helped us a lot to structurize the whole concept from our side. So in fact, there were a lot of great ideas they provided, guiding us...what we should do and what we should not do in this regard.And the best part was that they were very clear from there, from the whole blueprint. So they understood our request from the proposal and then accordingly they executed it.”
Hitesh MalhotraHitesh MalhotraSports Development at Fédération Internationale de Volleyball - FIVBSports
A Buyer's Choice Home Inspections and MACJ logo
“I've known PromotEdge for quite a few years. PromotEdge is passionate, knowledgeable, and destined for success. What stands out is their positivity, energy, passion, and innovative thinking. They bring a warm presence to their work. Best wishes to PromotEdge.”
Mahendra SurekaMahendra SurekaFounder & CEO at India MACJ - A Buyers Choice Home InspectionsReal estate
Birdman Chemeng Pvt Ltd.
“PromotEdge has been a unique creative group amongst unique creativity world. Approaching, analysing, understanding the deliverables with ethos and transferring the same into a spellbound creative identity, just cut out for clients’ requirement are signature values of the team. A huge ROI, thy name is PromotEdge.”
Rajat ChatterjeeRajat ChatterjeeHead Business Development at Birdman Chemeng Pvt Ltd.Manufacturing
Magik Lighting
“We are thoroughly impressed with the extensive knowledge and professionalism demonstrated by PromotEdge. Their adept management of our digital and creative requirements has significantly contributed to our satisfaction with their digital marketing services and expertise.”
Rahul MishraRahul MishraAssistant Manager Digital Marketing & Ecommerce at Magik LightingElectricals
Shyam Metalics Group
“We have been working with PromotEdge for almost a year. We have engaged them to manage our Digital marketing campaigns and are impressed with the outcome. Within weeks, we had noticed an increase in performance and they continuously worked on adapting to a new strategy if required to get the maximum mileage for us based on our requirements. A big thank you to their team, and I would highly recommend them to anyone who is considering growing on their Digital Space.”
Shubham GadodiaShubham GadodiaDeputy Brand Manager at Shyam Metalics GroupSteel and metals
Unimark Group
“As a progressive and emerging digital marketing company, PromotEdge is doing something right to enhance the experience of companies associated with them.The format they have created is great, there is a smooth flow of communication, great brainstorming sessions and additionally super ideations and executions.Pritha, the account manager, has always been supportive. Whether day or night, weekday or weekend, she came through and solved the problems.…”
Anish Basu MallikAnish Basu MallikMarketing Executive at Unimark GroupReal estate
Prime Volleyball League
“I would just like to congratulate, PromotEdge Led by Sayantan and his Team who have done an absolutely fantastic job of branding for the Kolkata Thunderbolts, the Branding, the social media campaign was bang on, and I think it help not just the Kolkata Thunderbolts but its really helped me because in the East there was at least one Team that looking very well promoted and branded. Thanks guyz, well done.”
Joy BhattacharjyaJoy BhattacharjyaCEO at Prime Volleyball LeagueSports
“In the city of Kolkata , there are very few full service advertising agencies which are compact, agile, quick and yet have the vision and creative /executional abilities of large reputed agencies. My experience with PromotEdge has been extremely satisfying and they have done world class job for Brand Building exercise of LION INDIA.In my professional life spanning over 4 decades I have always worked with leading agencies like Lintas, Grey, etc so my benchmarking was rather high. I am delighted to say that i did find the agency measuring up nicely on every parameter of our expectations. The team […] comes majorly from experienced staff who have had experience in large agencies. Added to this is the” young oozing with unique new.”
Hardeep SinghHardeep SinghMarketing Consultant
TATA Metaliks
“PromotEdge has been working with our organization for social media management and other creative activities. They are responsive, supportive and hard working. We look forward to a long and positive relationship with them.”
Medha SharmaMedha SharmaStrategic Head at TATA MetaliksSteel and metals
Nitya Gold & Diamonds
“I commend PromotEdge as a content and advertising partner. I applaud their top-notch team, swift problem-solving, and outstanding digital media work for my brand. I express my happiness and eagerness for a long-term partnership.”
Priyanka GuptaPriyanka GuptaDirector at Nitya Gold & DiamondsJewellery retail
Utkarsh India
“An aligned partner is always a welcoming step. Someone who can think deep through lines, understand the matrix of product management. Fortunate enough, PromotEdge has been a partner in progress for me. Their uniqueness lies in their hands down knowledge of delivering the strategy and communication needed for the hour. A practical approach to ensuring return of investment has been the day one mantra. An advisory to an executionist, the systematic approach by team PromotEdge has been the impeccable.”
Bobby SrinivasBobby SrinivasChief Manager Marketing & Brand Marketing at Utkarsh IndiaSteel and metals
PRAN-RFL Group logo
“PRAN & PromotEdge have been working together for last 1.5 years. Throughout, the work experience is good and satisfactory.As a client, we found PromotEdge very professional and dedicated towards any project assigned to them.Most importantly, They have a bunch of energetic young people who always think from clients’ point of view and come up with the solution which is appropriate for them.I would personally refer PromotEdge to those who want creative and contemporary work.We wish PromotEdge a successful journey ahead.”
Sharif Rezwan KabirSharif Rezwan KabirBrand Manager at PRAN RFL GroupFMCG
“It’s a great way to be back at PromotEdge. I had a great time working them in Kolkata Tunderbolts doing the Volley Biswas character and they all are like good friends like family Saurav Bhaiya, […], and as I speak, I see Sayantan da walking in. HA HA! But it’s been a great experience I am glad to again meeting them and being here at the PromotEdge. Make Calcutta relevant again.”
Priyam GhosePriyam GhoseContent Creator
“I had a great time in Kolkata, I met entire PromotEdge team and they are amazing. Design language that they follow is crazy, the copywriting is crazy. I think they are one of the best digital marketing agencies, not just in Kolkata, entire eastern region and I think brand should not leave the chance to interact with these guys, make stories together.So, if I had something here, I would have definitely collaborated with them and worked. Now is the time, I mean it’s not even a question whether digital is important or not you don’t have an option today. Go digital work with PromotEdge and create your brand stories. Thank You.”
Manish PandeyManish PandeyConsultant & Coach to Content Creator
“The PromotEdge Agency stay with us all these 3 days, since when we arrived..more, more than 3 days. Since in the airport, they stayed together with us (FIVB team).Sayantan and his team is providing you to see this video.I just (want to say) _Shukriya_. Thank you so much for your team, because we are here in front of the camera but all the team is behind. The entire team together did this great job so that you can see how much volleyball is a lovely sport.”
GibaGibaBrazilian Former Volleyball PlayerSports
Read all 20 testimonials
Talk to us

Want help with this?

Leave your name and number. A strategist calls you back in one working day.

  • One page, six answers.
  • Where the growth is, what to say, which channels.
  • A starting budget, quick wins, what to measure.
WhatsApp us

Free. No obligation. You can take the plan to another agency.

We use these details only to reply to you, by call or WhatsApp. Privacy policy.

Or call us: +91 98366 39955