November 2, 2026 · AI Search · AEO

How Bots Crawl, and How AEO Differs from SEO

Frame from the homepage crawl demo: a bot trail connecting tagged links, headings and text on seortp.com — A frame from the homepage crawl demo, slowed down so you can follow it.

For most of the last twenty years, "can the bots see our site?" meant one bot. Googlebot. Whole SEO programs were built around it: sitemaps, robots.txt, crawl budget, and a long conversation with developers every time the site moved to a new JavaScript framework.

Look at your server logs today and you will probably find a dozen other names. GPTBot. ClaudeBot. OAI-SearchBot. PerplexityBot. Some of them help decide whether your brand shows up when a buyer asks an AI assistant a question. Most of them behave differently from Googlebot, and most enterprise sites were tuned for Googlebot alone.

If you came here from the spider crawling my homepage, I owe you a small confession. Real crawlers don't walk. That was a slowed-down cartoon so you could watch it happen. A real crawl takes a fraction of a second, and the differences between the bots doing it are where SEO and AEO start to split.

How a search crawler works

Google documents its process in three stages: crawl, render, index.

First, Googlebot checks your robots.txt file. If a URL is blocked, it doesn't request it. If it's allowed, Googlebot downloads the page's HTML and pulls out every link it can find. Those links go into a queue of pages to visit next. That's how a crawler discovers your site, one link at a time.

Then the page waits to be rendered. Google opens it in a headless version of Chrome, runs the JavaScript, and reads the finished page the way a visitor would see it. Google says a page may sit in that queue for a few seconds, and sometimes longer. Only then does it index the rendered version.

That rendering step is why so many enterprise teams got away with heavy JavaScript sites. Google did the extra work. Developers built pages that only fill in after scripts run, and Googlebot came back and filled them in too.

Most AI crawlers skip that step

In December 2024, Giacomo Zecchini at MERJ and the team at Vercel published one of the clearest looks at how AI crawlers behave. None of the major AI crawlers they tested, including GPTBot, ClaudeBot and PerplexityBot, rendered JavaScript. The crawlers behind ChatGPT and Claude downloaded some JavaScript files and never ran them. Gemini was the exception, because it uses Google's own crawling setup.

In plain terms: if your product details, pricing or FAQ answers only appear after a script runs, many AI crawlers likely see a mostly empty page where your content should be.

The same study found that about a third of the requests from ChatGPT's and Claude's crawlers hit pages that returned a 404 error, compared with about 8% for Googlebot. These bots are newer and spend a lot of time on old or broken URLs. Redirects and clean sitemaps matter more than they used to.

That study, published in December 2024, is almost two years old, and these crawlers are changing fast. I would treat it as a reason to test your own pages, and not as a permanent rule.

One company, several bots

This is where a lot of enterprise teams got caught. In 2023, many IT and legal teams blocked "AI bots" across the board. It made sense at the time. The main worry was content being used to train models without permission.

The AI companies now run separate bots for separate jobs. OpenAI documents three: GPTBot collects content that may be used for training, OAI-SearchBot surfaces sites in ChatGPT's search results, and ChatGPT-User visits a page when a person asks ChatGPT to look at it. Anthropic has the same split with ClaudeBot, Claude-SearchBot and Claude-User.

OpenAI is direct about what blocking means. Sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers. Blocking GPTBot keeps your content out of training. Those are two different business decisions, and a blanket block makes both of them at once.

I can't name names, but this comes up in my conversations with marketing teams more often than you might think. Most of them don't know the block is there, because nobody has checked since it went in. The bots have changed since then. The rule hasn't. Those brands are probably missing from AI answers their buyers are already reading.

What an answer engine looks for

Getting crawled is only the first step. The bigger difference is what happens after.

A search engine ranks pages. An answer engine builds an answer and cites the pages it pulled from. Those turn out to be different lists.

Louise Linehan and Xibeijia Guan at Ahrefs ran 15,000 long-tail queries through Google and through ChatGPT, Gemini, Copilot and Perplexity. Only 12% of the links cited by ChatGPT, Gemini and Copilot ranked in Google's top 10 for the same prompt. Put another way, almost nine out of ten pages those assistants cited were not on page one of Google for that question. About 80% of the citations didn't rank anywhere in Google for the original query.

One likely reason is query fan-out. An assistant often doesn't search the exact question a person typed. It breaks the question into several smaller searches, pulls results for each one, and cites the pages that answer those pieces well. A page that answers one narrow part of the question directly can be cited even if it would never rank for the full question.

Google's own AI Overviews sit closer to classic rankings. In an earlier Ahrefs study, 76% of AI Overview citations came from pages in the top 10. So the gap between SEO and AEO depends partly on which answer engine you care about most.

What this means for your team

The question "can the bots see our site?" has quietly turned into three questions.

Can they get in? That is a robots.txt review, bot by bot, with IT, legal and brand in the room. Decide on training and on search separately.

Can they read it? Open one of your key pages and use View Source, not Inspect. Look for your main content in the raw HTML. If it isn't there, many AI crawlers probably can't see it either. To check at scale, run a site crawler twice, once with JavaScript rendering and once without. Site Audit in Ahrefs has an Execute JavaScript setting for exactly this. Test it on one section of the site before you change anything.

Is it worth quoting? This is where SEO and AEO split the most. Pages that state the answer plainly, close to the question, give an assistant something to lift.

The first two are mostly housekeeping, and fixing them helps Google too. The third is where a lot of search marketing will be decided over the next few years, and most teams are still early. That is a good position to be in, as long as you know which bots are reading your site and what they can see.

Sources

Written by Rob Delory. Questions? rob@seortp.com