Reading time: 13 minutes
A technical SEO audit checks whether Google, Bing and AI crawlers can crawl, read and index your site. In 2026, technical SEO also covers AI search crawlers, the CDN and the AI control in Search Console. So this checklist shows what to test, which tool to use and what drives the cost.
In short
- Technical SEO starts with Google’s three requirements: Googlebot has access, the page returns status 200 and the content is indexable.
- For AI, a technical SEO audit also checks AI search crawlers, the CDN and the firewall. After all, a block there can remove the site from answers that use search.
- The main text must be in the HTML the server sends, because not all crawlers run JavaScript, according to Google.
- Search Console, Bing Webmaster Tools and PageSpeed Insights are free. So the cost of the audit lies in the hours of analysis and the fix plan.
What goes into a technical SEO audit in 2026?
A technical SEO audit covers five blocks: crawler access, rendering, indexing, page signals and AI controls. The first three come from Google’s technical requirements. The last two, however, cover the practices and controls that apply to AI answers in Google and Bing.
According to Google Search’s technical requirements, a page needs three things: Googlebot not blocked, HTTP status 200 and indexable content. Even so, Google warns that it doesn’t guarantee indexing.
| Block | What to check | Where to check |
|---|---|---|
| Crawler access | robots.txt, CDN, firewall and HTTP status | Browser, curl and logs |
| Rendering | Text, links and price in the initial HTML | Source code and URL Inspection |
| Indexing | Key pages in the index, sitemap and canonical | Search Console and Bing Webmaster Tools |
| Page signals | Links with href, Core Web Vitals and structured data | PageSpeed Insights and Rich Results Test |
| AI controls | AI inclusion in Search Console, nosnippet and noarchive | Search Console and page code |
This checklist covers the technical base, while the SEO and AI guide covers the rest, from content to measurement. Finally, the free GEO audit measures brand presence in answers, with test questions and cited sources.
What changed in the SEO audit with AI?
In Google’s AI features, the bar is the same as in search. According to the page AI features and your website, AI Overviews and AI Mode have no extra technical requirements. In addition, its list of practices starts with access: crawling allowed in robots.txt and by any CDN or hosting.
So, compared with a technical SEO checklist from years ago, today’s audit adds four items:
- AI search crawlers: OAI-SearchBot, Claude-SearchBot and PerplexityBot bring the site into search-based answers in ChatGPT, Claude and Perplexity.
- CDN and firewall: in September 2026, Cloudflare widened two training options, Block and Block on pages with ads. They now also block Googlebot, Bingbot and Applebot.
- AI control in Search Console: in its guide to generative AI features, Google requires the site’s inclusion in those features. Inclusion is the default, but a child property inherits the choice of its parent property.
- Bing: the Bing index is the foundation of Copilot, so it joins the checklist next to Google.
On the other hand, special files don’t count for Google. The same guide says you don’t need llms.txt or other AI text files, because Google Search doesn’t use them. See what llms.txt is for in other services.
How do you check whether AI crawlers can access the site?
Start the technical SEO audit with robots.txt. No rule should block search crawlers, such as Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot. Then request key pages with each crawler’s user agent and compare the response codes. For example, a 403 or 429 for the crawler alone points to a CDN, firewall or hosting block.
In ChatGPT, OAI-SearchBot decides whether the site appears in search. According to OpenAI’s documentation, sites that block this crawler don’t appear in ChatGPT search answers. GPTBot, on the other hand, collects content for training, so blocking it is a separate decision. The guide on robots.txt for AI covers the choice crawler by crawler.
- Read the robots.txt file on each host. Look for
Disallow: /under a search crawler’s name or in aUser-agent: *group that closes the site. - Test the server response. Request robots.txt and the key pages with each crawler’s user agent and note the status.
- Open the CDN dashboard and the firewall. Bot rules there can stop an AI crawler even before it reads robots.txt.
- Check the logs. See whether the crawlers reach key pages and whether they get status 200.
How do you test the CDN, firewall and server in a technical audit?
The robots.txt file is a request, while the technical block sits in the infrastructure: CDN, firewall or hosting. Perplexity, for example, warns that with a web application firewall, you may need to explicitly allow its crawlers. So, on Cloudflare sites, check the AI policies and the Crawlers tab in AI Crawl Control. The steps are in the guide on how to check and allow AI crawlers on Cloudflare.
For technical SEO, Google also depends on a stable server. According to Search Console Help, Google slows or stops crawling when robots.txt doesn’t return a valid file or a 404. So check that the server delivers the file without errors to every crawler.
Finally, check the IP, since anyone can fake a user agent. OpenAI, Anthropic and Perplexity publish IP lists for their crawlers.
How do you test whether content depends on JavaScript?
In this part of the technical SEO audit, compare the page’s source code with what the browser shows. If the main text, links or price appear only after JavaScript runs, a crawler that skips the code misses them. Googlebot renders pages, but Google itself warns that not all crawlers do.
In its JavaScript SEO basics, Google calls server-side rendering or pre-rendering a great idea. After all, it makes the site faster, and not all bots can run JavaScript. Likewise, the Bing guidelines, which also apply to Copilot, say not to hide critical content behind client-side rendering. According to Bing, content that doesn’t render reliably may stay out of the index and out of answers.
On the other hand, OpenAI, Anthropic and Perplexity don’t say, on their crawler pages, whether their crawlers run JavaScript. So treat the initial HTML as what those crawlers read.
- Source code without JavaScript: open the page source or download the HTML with curl. Then search for a sentence from the text and for the links.
- URL Inspection: in Search Console, run the live test and open “View tested page,” which shows the screenshot and the HTML.
- Internal links: check that they are
<a>elements with href, the format Google can crawl.
What should you do when content only exists in JavaScript?
In technical SEO, the fix generally depends on the size of the problem. When the HTML lacks only links, meta tags or structured data, changes to the current code solve it. The JavaScript SEO checklist shows how. If all the main text only appears in the browser, however, the answer is to switch the rendering mode. In that case, the comparison of SSR and CSR helps you choose between server rendering, static pages and pre-rendering.
How do you check indexing in Google and Bing?
In Google, the technical SEO audit uses URL Inspection on key pages and the Page indexing report for the site. Likewise, in Bing Webmaster Tools, use URL Inspection and Site Explorer. So the two free dashboards show whether a page is in the index, why it’s missing and what to fix.
- Key pages in the index: most sites have some pages outside the index, says Search Console Help. However, all important pages need to be in it, and duplicates shouldn’t.
- Sitemap: submit it to both search engines, with canonical URLs only, as the Bing guidelines ask.
- noindex in the right place: robots.txt controls crawling, not indexing. To remove a page from the index, use noindex and let the crawler fetch the URL.
- Snippets allowed: for Google’s AI features, a page must be eligible to show with a snippet. So look for nosnippet on pages that AI should cite.
What does a technical SEO audit check in Bing?
Bing matters for AI in two ways. First, it’s the foundation of Copilot. In addition, on the Bing Webmaster blog, Microsoft says Bing powers AI tools like ChatGPT. So check that nothing blocks Bingbot, that Bing processed the sitemap and that IndexNow reports changes.
Also look for noarchive and nocache on key pages. According to the Bing guidelines, the first keeps content out of Copilot answers, and the second limits what Copilot can use. The guide on how to index your site on Bing covers sign-up, both directives and the role of Bing in ChatGPT.
Which tools should you use for a technical SEO audit?
Free official tools cover almost the whole technical SEO audit. They are Search Console, Bing Webmaster Tools, PageSpeed Insights and the Rich Results Test. In addition, the browser and curl show the HTML that a crawler without JavaScript gets. Finally, server or CDN logs show who visited the site and with what status.
| Tool | Use in the audit | Cost |
|---|---|---|
| Search Console | Index, URL Inspection, robots.txt and crawling | Free |
| Bing Webmaster Tools | Bing index, Site Scan, IndexNow and Copilot citations | Free |
| PageSpeed Insights | Core Web Vitals for each page | Free |
| Rich Results Test | Structured data and rendered HTML | Free |
| Browser and curl | Initial HTML by user agent | Free |
Bing even has its own technical SEO auditor. According to Bing Webmaster Tools help, Site Scan crawls the site on demand and checks for common technical issues. The report splits errors, warnings and notices, and the help says to fix errors first, because they can affect indexing.
Paid third-party tools also help, but with a caveat. In its guidance on third-party tools, Google says they can’t access its internal ranking data or guarantee performance. So it recommends Search Console even to people who use another tool. For Core Web Vitals, Google recommends LCP within 2.5 seconds, INP under 200 milliseconds and CLS under 0.1.
Example: a WordPress blog on the technical SEO checklist
At NotHuman, the blog runs on WordPress, and the text of each article already comes in the initial HTML. We checked this with requests without JavaScript, as an AI crawler does. The robots.txt file doesn’t block GPTBot, Google-Extended or CCBot, as a GEO decision. In addition, Yoast publishes the automatic llms.txt, notifies IndexNow and generates the BlogPosting and BreadcrumbList schema. It adds FAQPage only to articles that have FAQs.
How much does a technical SEO audit cost?
We found no reliable price benchmark for Brazil. No survey with a stated sample turned up. The published ranges come from agencies or from price tables with no stated method. So, in practice, a technical SEO audit costs hours of analysis, and four factors drive those hours up.
- Size and page templates: a corporate site needs fewer hours than a store with filters and thousands of URLs.
- JavaScript: on sites built with React, Vue or Angular, technical SEO calls for rendering tests on each page template.
- Access: logs, CDN and firewall need someone from the client’s IT team.
- Languages and countries: each local version multiplies the indexing and hreflang checks.
What should an SEO audit deliver?
In Do you need an SEO?, Google says an SEO audit should give realistic estimates of improvement and of the work involved. The page also warns against anyone who guarantees first place. Google even recommends giving the auditor only read access to Search Console at this stage.
In practice, ask for five items in the report:
- Problems found, with sample URLs and evidence, such as a screenshot and the response code.
- Priority for each item, with crawler blocks and key pages missing from the index at the top.
- Suggested fix, with an owner: IT, content or SEO.
- Effort estimate for each fix.
- Retest after the fixes, with the same tools.
For an agency, the technical SEO audit becomes the client’s entry diagnosis. So it pays to charge for it as a fixed project, apart from the monthly scope. The guide on GEO for agencies recommends the same for the GEO diagnosis.
What we found when we analyzed 10 technical SEO audit guides
We ran four searches and gathered 10 Brazilian Portuguese guides that teach how to do a technical SEO audit. All come from agencies, consultants or SEO tools. In addition, seven carry a 2026 date, either for publication or for an update.
- AI crawlers: none mentions GPTBot, OAI-SearchBot or any other AI crawler.
- Bing and CDN: none mentions Bing, a CDN or a firewall.
- JavaScript: three say to test rendering, always with a focus on Google and search engines.
- Tools: eight mention Search Console, and nine talk about Core Web Vitals.
Frequently asked questions
No. A technical SEO audit checks crawler access, rendering, indexing and page signals. A full SEO audit, on the other hand, adds content, keywords and links from other sites. So technical SEO usually comes first: without access and indexing, the rest doesn’t show up in search or AI.
Not for Google. According to the Google guide to generative AI features, Google Search doesn’t use llms.txt or other AI text files. Keeping the file for other services isn’t a problem, as the guide on what llms.txt is for explains.
It isn’t required. According to the Google guide to generative AI features, those features don’t need structured data. Even so, Google recommends keeping it, because it helps pages qualify for rich results. In the audit, also check that the markup matches the visible text.
Redo the technical SEO audit before and after big changes, such as a redesign, a migration or a CDN switch. Google also points to a redesign and a site launch as good times to hire an SEO. Finally, between audits, keep an eye on Search Console and Bing Webmaster Tools.
Want to know how your brand shows up in AI answers today?
NotHuman is opening spots for agencies and brands to track how they appear in AI answers. Join the waitlist and we’ll let you know when a spot opens up for you.
Main sources
- Google Search Central, Google Search technical requirements (2025).
- Google Search Central, AI features and your website (2025).
- Google Search Central, Optimizing your website for generative AI features on Google Search (2026).
- Google Search Central, Understand the JavaScript SEO basics (2026).
- Google Search Central, Do you need an SEO? (2026).
- OpenAI, Overview of OpenAI Crawlers (2026).
- Bing Webmaster Tools, Bing Webmaster Guidelines (2026).
- Bing Webmaster Blog, Start Using Bing Webmaster Tools to Improve Your Site Visibility (2025).
