Robots.txt and Meta Robots: A Crawler Guide

There are two levers for controlling what search engines do with your site, and they control different machines. Robots.txt controls crawling — whether a bot may fetch a URL at all. Meta robots (and its HTTP-header sibling, X-Robots-Tag) controls indexing — whether a fetched page may appear in search results. They sound interchangeable. They are not, and mixing them up produces the single most common self-inflicted wound in technical SEO: pages you tried to hide that show up in Google anyway, or pages you needed indexed that silently vanish.

This guide covers what each mechanism actually does, the interaction trap between them, the directives worth knowing, a sane default setup for WordPress, and the newer question of AI crawlers — which robots.txt has suddenly become the front line for.

The Core Distinction, Stated Plainly

The counterintuitive row is the fourth one. A robots.txt Disallow says "don't fetch this," and Google complies — but if other pages link to that URL, Google can still index the address without its content, producing the ghostly result: your URL with the description "No information is available for this page." Robots.txt hides content; it does not hide the existence of URLs.

The Trap: Disallow + Noindex = Noindex Never Seen

Here's the interaction that catches even experienced people. Suppose you want /members/ out of Google. You add a noindex meta tag to the pages and, for good measure, block /members/ in robots.txt. Belt and suspenders? No — you've just made the noindex invisible. Google can't fetch the pages (robots.txt forbids it), so it never sees the noindex tag, so the URLs can remain indexed as ghosts indefinitely.

The rule to tattoo somewhere: for a page to be de-indexed by noindex, it must remain crawlable. The correct sequences:

• Get pages out of the index: noindex them, leave them crawlable, wait for recrawl (days to weeks). Only after they're gone may you optionally block crawling too.
• Stop crawl waste on junk URLs at scale: robots.txt alone. Accept that some ghost URLs may linger in the index; they rarely surface for real queries.
• Truly private content: neither tool. Robots.txt is a public file readable by anyone (including attackers scouting your interesting paths), and noindex is a request, not a lock. Private means authentication.

Robots.txt: Syntax and Sharp Edges

The file is a set of rule groups, each targeting a user agent. The directives that matter:

• User-agent: which bot the group applies to (* for all, Googlebot, Bingbot, GPTBot...). A bot obeys the most specific group that names it, ignoring the * group entirely in that case — a classic gotcha: if you create a "User-agent: Googlebot" group, Googlebot no longer reads your * rules at all.
• Disallow: a path prefix not to fetch. Disallow: /wp-admin/ blocks everything under that path. An empty Disallow: means "nothing blocked."
• Allow: an exception carved out of a Disallow — e.g., block /wp-admin/ but Allow: /wp-admin/admin-ajax.php (the standard WordPress pattern, since some front-end features call that endpoint). The more specific (longer) rule wins.
• Wildcards: * matches any characters, $ anchors the end. Disallow: /*?sort= kills sort-parameter crawling; Disallow: /*.pdf$ blocks PDFs.
• Sitemap: the absolute URL of your sitemap index. Include it — it's how crawlers you never submitted to find your sitemap.

Sharp edges worth respecting:

1. An error in robots.txt can deindex your site. A stray Disallow: / blocks everything. This is the most catastrophic single character in web operations. Check the file after every migration and any security-plugin change.
2. If robots.txt itself returns a 5xx error, Google may treat the whole site as disallowed until it can read the file again. (A 404 is fine — no file means no restrictions.)
3. Never block CSS and JavaScript. Google renders pages; blocking /wp-includes/ or asset paths (an old "optimization") makes your pages render broken in Google's eyes and hurts rankings. Modern default: leave assets crawlable.
4. Case sensitivity and precision: paths are case-sensitive; Disallow: /Private/ does not block /private/. And remember these are prefix matches — Disallow: /blog also blocks /blogging-tips.
5. Compliance is voluntary. Reputable crawlers obey; scrapers and malicious bots read robots.txt as a menu. It's a traffic sign, not a wall.

Meta Robots and X-Robots-Tag: The Directives That Matter

The meta tag lives in the page head — meta name="robots" content="noindex, nofollow" — and its useful values:

• noindex: don't show this page in results. The workhorse.
• nofollow: don't follow this page's links (rarely what you want site-internally — it wastes the page's ability to pass signals; use sparingly).
• noindex, follow: the classic archive setting — keep the page out of results but let its links be discovered. Note Google eventually treats long-standing noindex pages' links as nofollow anyway, so don't build architecture that depends on noindexed hub pages.
• noarchive, nosnippet, max-snippet, max-image-preview: presentation controls — how much of your content appears in the result. Publishers negotiating content visibility use these; most sites can ignore them.
• The default is index, follow — you never need a tag saying so, despite a decade of themes emitting it.

X-Robots-Tag is the same vocabulary delivered as an HTTP header, which is the only way to noindex non-HTML files — PDFs, images, feeds. Set it at the server level (an Apache/nginx rule adding "X-Robots-Tag: noindex" to PDF responses, say) when you need documents kept out of results without deleting them.

A Sane WordPress Setup

WordPress serves a virtual robots.txt by default; SEO plugins let you edit it. A defensible baseline for a typical site:

• User-agent: * with Disallow: /wp-admin/ plus Allow: /wp-admin/admin-ajax.php — the long-standing standard.
• Disallow internal search: Disallow: /?s= and Disallow: /search/ — internal search results are the canonical crawl-trap, and Google recommends keeping them uncrawled.
• If you run WooCommerce: block the non-content flows — Disallow: /cart/, /checkout/, /my-account/ — plus any add-to-cart parameters (Disallow: /*?add-to-cart=).
• Facet/sort parameters if applicable: Disallow: /*?orderby= and friends — check your crawl stats for which parameters bots actually chew on.
• Sitemap: line pointing at your sitemap index.

And the indexing side, set in your SEO plugin (these become meta robots tags, not robots.txt entries):

• Noindex: internal search results (defense in depth), attachment pages (or redirect them), date archives, and usually tag archives unless yours are curated and substantial.
• Author archives: noindex on single-author sites (it's a duplicate of the blog index), index on genuine multi-author publications.
• Keep indexed: posts, pages, products, categories that act as real landing pages.

One thing you should not do in WordPress: hand-edit a physical robots.txt while an SEO plugin also manages a virtual one — you'll get confusion about which is served. Pick one mechanism.

The New Front: AI Crawlers

Robots.txt has acquired a second job: expressing policy toward AI training and answer engines. The major AI operators publish user agents — GPTBot (OpenAI training), ClaudeBot (Anthropic), Google-Extended (Google's AI training switch, separate from Googlebot), PerplexityBot, CCBot (Common Crawl) — and honor Disallow rules for them.

The decision is genuinely strategic, not technical: blocking training bots keeps your content out of future model training (prospectively — it doesn't unlearn anything), while blocking retrieval-oriented bots can remove you from AI answers that increasingly carry referral traffic. There's no universally right answer; a reasonable stance for most small businesses is to leave retrieval/search bots alone (visibility is the business) and decide on training bots per your comfort. What matters mechanically: these are just user-agent groups in the same file, and remember the specificity gotcha — giving GPTBot its own group means it ignores your * rules, so repeat any general Disallows inside its group.

Testing: Never Deploy Blind

1. Search Console's robots.txt report (Settings → robots.txt) shows the fetched file, fetch errors, and parse warnings.
2. URL Inspection on a sample of important pages after any change: it reports both "Crawl allowed?" and "Indexing allowed?" — the two questions this whole article is about, answered per URL.
3. Crawl your own site with Screaming Frog respecting robots.txt, and diff the crawlable set before and after a change. The pages that disappeared are your blast radius.
4. After launching a site: confirm the WordPress "Discourage search engines" checkbox (Settings → Reading) is off. It sets sitewide noindex and is the most common reason a new site is invisible for months.

The Bottom Line

Two levers, two jobs. Robots.txt manages crawler effort — point it at the junk URL patterns that waste fetches, keep it away from CSS, JS, and anything you want de-indexed, and treat every edit as production-critical. Meta robots (and X-Robots-Tag for files) manages visibility — noindex the real pages that shouldn't rank, and leave them crawlable so the instruction can be read. Keep the two consistent, test with URL Inspection after changes, and revisit the file once a year — it now encodes not just your crawl policy but your stance on AI, and both deserve to be deliberate.

Photo: Pavel Danilyuk on Pexels