robots Meta Tag's 4 Combinations — Why noindex Can Still follow
The first time most people encounter <meta name="robots" content="...">, they assume "noindex just blocks everything, right?" But this tag's value isn't a single switch — it's actually two switches that operate completely independently of each other. Without understanding this structure, you can't handle the very common requirement of "I want to block indexing but keep the link structure intact," and you end up applying an unnecessarily total block that severs your own site's link graph.
1. index/noindex and follow/nofollow answer different questions
The robots meta tag's two axes address completely different questions. index/noindex decides "is this page itself allowed to show up in search results?", while follow/nofollow decides "are the links on this page allowed to be followed for crawling other pages?" Since these are two fully independent binary values rather than one being a precondition for the other, all 2×2 = 4 combinations are mathematically valid and all four are actually used.
| Combination | Shown in search results? | Links on this page followed for crawling? |
|---|---|---|
index, follow (default) | Shown | Followed |
noindex, follow | Not shown | Followed |
index, nofollow | Shown | Not followed |
noindex, nofollow | Not shown | Not followed |
2. When you actually need noindex,follow
The most practically useful combination is noindex, follow. It's used for pages like tag pages, pagination beyond page 2, or filter-combination URLs — pages that are "thin content on their own and don't need to appear individually in search results, but whose links to products or posts still need to be continuously discovered by crawlers." If you instead apply noindex, nofollow to such a page, the crawler can never discover any sub-links that are only reachable through it, and part of your site unintentionally becomes a crawling blind spot. Running the Meta Tag Analyzer on an actual page lets you immediately check whether this combination is applied correctly on your category or tag pages.
3. The rarer opposite case: when index,nofollow is used
index, nofollow means "I want this page itself to appear in search, but I don't trust the links leaving this page, so don't follow them." It's used for cases like user-submitted content (reviews, comments, forum posts), where the page body itself is valuable but the outbound links inside it might be spam. In practice, though, applying rel="nofollow" to individual links is more granular and far more common than blocking at the page-level meta tag, so meta-tag-level index,nofollow is relatively rare.
4. What does the actual checking tool verify?
The Meta Tag Analyzer works by extracting the meta[name="robots"] value straight from the page HTML and displaying it as-is. That means if this combination is mistakenly set incorrectly (e.g. a leftover noindex from a staging deploy), the value itself will still show up exactly as it is — so you always need to verify the actual rendered HTML before deploying. It's also worth remembering that if noindex is only set via a server response header (X-Robots-Tag) rather than in the HTML source, checking the HTML alone can miss it.
<meta name="robots" content="noindex,nofollow"> before pushing it to production. Since the site looks completely normal visually, this kind of mistake is hard to spot with the naked eye, and it's often discovered late — only after "Excluded by 'noindex' tag" entries start piling up in Search Console's indexing report.
5. The difference from robots.txt — and why you shouldn't mix the two mechanisms
robots.txt blocks "crawling (fetching) itself," while the robots meta tag's noindex means "crawl it, but exclude it from the index." If you block a path via robots.txt, the crawler can never access that page, and so it never even gets to read the noindex meta tag inside it. As a result, if that page already had links pointing to it from elsewhere, you can get the opposite of what you wanted — a bare URL with no content sitting in search results. So if your actual goal is "keep this completely out of search," don't block it with robots.txt — use noindex alone, so the crawler can actually read and act on that instruction.
Frequently Asked Questions
Q. What happens if I don't set a value at all?
A. If there's no robots meta tag at all, or its value is empty, the default index, follow applies. In other words, "allow search indexing + allow following links" is the web's default behavior.
Q. Do other search engines besides Google interpret these 4 combinations the same way?
A. Major search engines like Bing and Naver also follow the same basic meaning of index/noindex and follow/nofollow. However, support for finer engine-specific extension values (noarchive, nosnippet, etc.) can vary by search engine, so it's safest to also check each engine's official documentation.
Q. Does a page disappear from search results immediately after adding noindex?
A. Not immediately. A crawler has to revisit the page and read the new meta tag value before the change takes effect, so depending on crawl frequency it can take anywhere from a few days to a few weeks. If you need faster removal, it's a good idea to also use the URL removal tool in Search Console.
Q. Should tag pages always have noindex,follow applied?
A. There's no one-size-fits-all answer. If a tag page aggregates sufficiently unique content and there's real search demand for it, keeping it indexed can be the better call. The general rule of thumb is to apply noindex,follow only when you have a lot of low-quality, highly duplicative tag pages to clean up.