What is indexing?
Indexing is the process in which a search engine analyzes the pages it has crawled and stores them in its index, the database it searches whenever someone enters a query. A page outside the index cannot appear in search results, however good it is.
The search engine first discovers a URL through links or an XML sitemap, then crawls and renders the page. Finally it decides whether to store the page and, for near-identical pages, which URL to keep as the canonical version. Duplicates, pages with little unique content and pages with a noindex tag are usually left out.
Crawling and indexing are different: robots.txt controls what crawlers may fetch, while a noindex tag controls what may be indexed. Google also says it does not guarantee indexing, even for pages that follow its guidelines.
Why indexing matters for AI visibility
AI features and assistants that search the web look up candidate pages in a search index and write their answer from those, a method known as retrieval-augmented generation. Google says a page must be indexed and eligible to be shown with a snippet in Search before it can appear as a supporting link in AI Overviews or AI Mode. Other assistants use their own search crawlers, results from search engines or both. Microsoft Copilot, for example, draws on Bing.
A page that is not indexed therefore cannot be retrieved or cited. Indexing also sets the pace: a new pricing page only reaches AI answers once it is indexed, and until then assistants rely on older sources. Being indexed makes a page eligible, not chosen.
How to check and improve indexing
In Google Search Console, the URL Inspection tool shows whether a page is indexed, and the Page indexing report explains why pages are excluded. Bing Webmaster Tools offers similar reports.
Submit an XML sitemap that lists only canonical, indexable URLs.
Link important pages from your navigation and related content; pages without internal links are hard to discover.
Remove accidental noindex tags from pages you want found, and do not block those pages in robots.txt.
Serve your main content in the initial HTML, for example with server-side rendering, and point duplicates to the preferred URL with canonical tags.
Use IndexNow, a protocol supported by Bing and several other search engines, to report new and changed URLs right away. Google does not support it.
Trackbase Crawler Analytics shows which AI crawlers actually visit your site and what they read.
Examples of indexing
A B2B payroll software vendor's new pricing page sits at "Discovered - currently not indexed" for weeks: the sitemap lists it, but no other page links to it. Once the navigation and feature pages link to it, it gets indexed and can be cited in AI Overviews.
A Dutch bicycle webshop finds thousands of filter URLs in the index while some product pages are missing. It adds canonical tags and limits its sitemap to product and category pages.
A SaaS company's help center is well indexed in Google but largely missing from Bing, so the team submits its sitemap to Bing Webmaster Tools and starts using IndexNow.
A marketing agency adds index status to its monthly client reports, so it can rule out indexing problems before analyzing a drop in AI citations.
Frequently asked questions
What is the difference between crawling and indexing?
Crawling is fetching a page; indexing is analyzing it and storing it so it can appear in results. A page can be crawled and still not be indexed, for example when it is a duplicate or has a noindex tag. robots.txt controls crawling and a noindex tag controls indexing. To keep a page out of the index, use noindex and let crawlers fetch the page, or they never see the tag.
How do I check whether my pages are indexed?
In Google Search Console, enter a URL in the URL Inspection tool to see its status, or open the Page indexing report for the whole site, including the reasons pages are excluded. Bing Webmaster Tools offers similar reports for Bing. A search with the site: operator in Google gives a rough impression, but it is not a complete or exact list.
Does being indexed mean AI assistants will cite my page?
No. Indexing only makes a page eligible. AI features and assistants choose from many indexed pages, based on how well each one answers the question and how trustworthy the source seems. Being indexed is the entry ticket; relevance, clear answers, authority and mentions on other sites decide whether your page is actually used and cited.