Meta Robots Tag Guide: Control Search Indexing

The meta robots tag is one of the most powerful and most misunderstood tools in technical SEO. It tells search engines what to do with a page: whether to include it in search results, whether to follow its links, whether to cache the content, and more.

Get it right and you have precise control over which pages appear in Google, Bing, and other search engines. Get it wrong and you can accidentally de-index your entire site.

The tag goes in the section of your HTML:

`html `

This default directive tells search engines to index the page and follow all links on it. Most pages should have this (or no meta robots tag at all, since index/follow is the default behavior).

Understanding Each Directive

index: allow the page to appear in search results. This is the default.

noindex: prevent the page from appearing in search results. The page can still be crawled, but it will not show up when someone searches. Use this for thank-you pages, internal search results, admin pages, and staging environments.

follow: allow the search engine to follow links on this page and pass link equity. This is the default.

nofollow: tell the search engine not to follow links on this page. This does not prevent the page from being indexed; it only affects outbound links. Use this for user-generated content pages where you cannot vouch for the quality of linked sites.

noarchive: prevent search engines from showing a cached version of the page. Useful for pages with time-sensitive information or content you want to control access to.

nosnippet: prevent search engines from showing any text snippet or video preview in search results. The page still appears but with only the title and URL.

max-snippet:[number]: limit the text snippet to a specific number of characters. max-snippet:0 is equivalent to nosnippet. max-snippet:100 limits the snippet to 100 characters.

max-image-preview:[size]: control the size of image previews. Options are none, standard, or large. For most sites, large is best because bigger previews get more clicks.

Pair your robots tag with a clean set of head tags using the Meta Tag Generator to control titles, descriptions, and social previews alongside indexing.

Common Meta Robots Mistakes

Noindex on important pages: the most devastating mistake. If your homepage, product pages, or key landing pages have noindex, they disappear from search results. Always check for accidental noindex tags after a site migration or CMS update.

Noindex on staging that leaks to production: staging environments should have noindex. But when you copy code from staging to production, the noindex tag can come along. Set up monitoring to alert you if production pages have unexpected noindex tags.

Using nofollow to sculpt PageRank: this was a tactic in the early 2010s. It no longer works. Google still calculates the PageRank flow but throws away the portion that would go to nofollowed links instead of redistributing it to followed links.

Robots tag on robots.txt-blocked pages: if robots.txt blocks a page from being crawled, search engines never see the meta robots tag. They might still index the URL based on external links pointing to it. To prevent indexing, the page must be crawlable so the search engine can read the noindex directive.

Conflicting directives: if your says index but your HTTP X-Robots-Tag header says noindex, the most restrictive directive wins. The page will not be indexed. Check both locations.

Use the Slug Generator to create clean, SEO-friendly URLs for pages you do want indexed. A good URL structure helps search engines understand your site hierarchy.

When to Use Noindex

Pages that should typically have noindex:

Internal search results pages: these create near-infinite URL variations with thin content. Google specifically warns against indexing them.

Paginated archives beyond page 1: the first page of a blog archive should be indexed. Pages 2, 3, 4, and beyond are typically thin and duplicative. Use noindex, follow so search engines still discover the articles linked from later pages.

Thank-you and confirmation pages: "Thank you for subscribing" pages have no search value and can dilute your site's quality signals.

Parameter-based URLs: URLs with sorting, filtering, or session parameters (e.g., ?sort=price&color=red) create duplicate content. Use noindex or canonical tags to consolidate.

Staging and development environments: always noindex to prevent accidental indexing of incomplete or test content.

Login-protected content: if the content is behind a login, indexing the login page with a snippet of the protected content is usually undesirable.

Legal boilerplate: privacy policy, terms of service, and cookie policy pages. These need to exist but rarely provide search value. However, some SEOs argue these pages build trust signals and should be indexed. Test for your specific situation.

For URLs with special characters or encoded parameters, the URL Encoder ensures they are properly formatted for both browsers and search engines.

X-Robots-Tag HTTP Header vs Meta Tag

The meta tag goes in the HTML. The X-Robots-Tag is an HTTP response header that does the same thing but works for any file type, not just HTML.

` X-Robots-Tag: noindex, nofollow `

When to use the HTTP header instead of the meta tag:

PDF files: you cannot put a meta tag in a PDF. Use the X-Robots-Tag header to prevent PDF files from being indexed if needed.

Image files: same reasoning. If you do not want Google Images to index certain images, use the header.

API responses: prevent JSON or XML endpoints from being indexed.

Bulk application: setting the header at the server or CDN level can apply to thousands of URLs without modifying each page's HTML. Useful for entire directories or file types.

Configuration in common web servers:

`nginx # Nginx - noindex all PDF files location ~* \.pdf$ { add_header X-Robots-Tag "noindex"; } `

`apache # Apache - noindex staging subdomain Header set X-Robots-Tag "noindex" `

Both the meta tag and the HTTP header are equally respected by search engines. Use whichever is easier to implement for your specific situation.

Testing and Validating Your Robots Directives

After setting up meta robots tags, verify they are working correctly:

Google Search Console URL Inspection: paste any URL from your site and see how Google interprets the robots directives. This shows the actual indexing status, not just what you intended.

View page source: the simplest check. Open the page, view source (Ctrl+U), and search for robots. Verify the content attribute matches your intent.

HTTP header check: use browser DevTools (Network tab) or curl -I to verify X-Robots-Tag headers on non-HTML resources.

Site audit tools: Screaming Frog, Ahrefs, and Sitebulb can crawl your entire site and flag pages with unexpected robots directives. This catches issues at scale that manual checks miss.

Google search operator: search site:yourdomain.com to see which pages Google has indexed. If an important page is missing, check its robots tag. If an unimportant page is present, add noindex.

Monitor after changes: after any site migration, CMS update, or large-scale template change, re-audit your robots directives. These are the events most likely to introduce accidental noindex tags.

Remember that it takes days to weeks for Google to process robots directive changes. A page you just noindexed will not disappear from search results immediately. A page you just re-indexed will not appear immediately either.

FAQ

Is the meta robots tag the same as robots.txt?

No. robots.txt controls which pages search engines can crawl (visit). The meta robots tag controls what search engines do with pages they have already crawled (index or not, follow links or not). They serve different purposes and should be used together.

Does noindex remove a page from Google immediately?

No. It can take days to weeks for Google to re-crawl the page and process the noindex directive. For faster removal, use the URL Removal tool in Google Search Console, which provides temporary removal within hours while the noindex takes permanent effect.

Should I use both noindex and nofollow together?

Not usually. If you want a page removed from search results but still want search engines to discover pages linked from it, use noindex, follow. Only add nofollow if you also want to prevent link equity from flowing through the page. The most common combination for pages you want to hide is noindex, follow.

Do social media crawlers respect meta robots tags?

It depends on the platform. Google and Bing respect the standard robots meta tag. Facebook, Twitter, and LinkedIn have their own crawlers that typically ignore noindex for the purpose of generating link previews. To control social media previews, use Open Graph and Twitter Card meta tags instead.