--- title: The noindex you cannot see: X-Robots-Tag and disappearing pages description: A noindex sent as an HTTP header renders nothing in the page source. How to find it and why pages vanish without explanation. url: https://moxseo.com/x-robots-tag-noindex date_modified: 2026-07-29 author: Ashish Khan language: en_US --- A well-written and properly linked page can still disappear from Google without any visible explanation in its HTML. One common reason is an X-Robots-Tag noindex instruction delivered through the HTTP response header instead of a standard meta tag. Because it does not appear in view-source, the page may look completely indexable while search engines are being told to remove it from the index. ## Two ways to say noindex There are two mechanisms, and they carry equal weight. The first is the familiar meta tag in the page head. The second is an `X-Robots-Tag` HTTP response header, sent by the server before any HTML arrives. The difference matters because only one of them is visible when you look at a page. Right-click, view source, search for “noindex”, find nothing, conclude the page is indexable — and be wrong. - **Meta robots** — sits in the HTML head, visible in view-source - **X-Robots-Tag** — sits in the HTTP response headers, invisible in view-source - Both are honoured. Either alone is enough to remove a page from the index. ## Where header-level noindex usually comes from It is rarely deliberate. In our experience it arrives from one of a handful of places, none of which announce themselves. - A staging environment promoted to production with its server config intact - A security or performance plugin applying rules to a directory pattern - A CDN or reverse proxy rule added for one path and matching more than intended - A file-type rule, for example applied to PDFs, that catches HTML too The common thread is that the rule lives in server configuration rather than in the site, so nobody editing pages ever sees it. We hit a version of this while building our own audit tool. Our first implementation only read the robots meta tag, which meant a page carrying a header-level noindex would have been reported as fully indexable. We caught it against a test page built specifically to lie to us, and it is the reason the tool now reads response headers as well as page source. ## How to check You need to look at response headers, not page source. Our [on-page audit tool](https://moxseo.com/tools/on-page-audit/) reads both and reports whichever is present, so a header-level noindex is named explicitly rather than silently missed. If you prefer to check manually, any HTTP inspection will do — browser dev tools under the Network tab, or a command-line request. What you are looking for is an `X-Robots-Tag` line in the response. Check the specific URL that is missing, not the homepage. These rules are almost always scoped to a path, which is exactly why they go unnoticed. ## Why the page may still appear indexed Removal is not instant. A page carrying a fresh noindex can remain in the index for days or weeks, and a `site:` search will still return it during that window. That lag is the reason this problem often goes undiagnosed. By the time traffic drops enough to investigate, the change that caused it is weeks old and nobody connects the two. If a tool reports a noindex on a page you can still find in search, believe the tool. The page is on its way out. ## How to actually read your response headers ![Three-step process for checking X-Robots-Tag directives through browser developer tools, command-line headers, and an SEO audit tool.](https://moxseo.com/wp-content/uploads/2026/07/check-x-robots-tag-response-headers-1024x683.webp)Image Source: AI-generated visual by MoxSEO You cannot see this in view-source, so the check has to happen at the HTTP layer. There are three practical routes, in rough order of convenience. - **Browser dev tools.** Open the Network tab, reload, click the first document request, read the Response Headers panel. Look for a line beginning `x-robots-tag`. - **Command line.** A single request with headers returned is enough. You are reading the response, not the body. - **A tool that checks both.** Our [on-page audit](https://moxseo.com/tools/on-page-audit/) reports whichever mechanism is present and names it explicitly. Whichever route you take, check the specific URL that is missing rather than the homepage. These rules are almost always scoped to a path, which is precisely why they survive undetected. ## The syntax, and the part that catches people out A header can target all crawlers or one in particular. Both of these are valid: - `X-Robots-Tag: noindex` — applies to every crawler that honours the protocol - `X-Robots-Tag: googlebot: noindex` — applies to Googlebot only The second form is the one that produces genuinely confusing symptoms. The page performs normally in Bing, is absent from Google, and nothing in the HTML explains the difference. A page can also carry several X-Robots-Tag lines, each targeting a different crawler. If you find one, keep reading — there may be more below it. ## Fixing it, and what to expect afterwards Find the rule and remove it at source. That will be in your server configuration, a plugin setting, or a CDN rule, depending on where it came from. Editing the page will do nothing. Once removed, request indexing for the affected URLs in Search Console rather than waiting. A page that was deliberately excluded may not be recrawled promptly on its own. Expect days rather than hours, and expect the recovery to be uneven. Pages with stronger internal linking tend to return first, which is a reasonable proxy for how your own site ranks their importance. One caution: if the noindex was applied deliberately by someone else, find out why before removing it. Staging environments, client-only areas and thin archive pages are all legitimately excluded, and undoing that creates a different problem. ## Sources and further reading - [Robots meta tag, data-nosnippet and X-Robots-Tag specifications](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag) - [Block search indexing with noindex](https://developers.google.com/search/docs/crawling-indexing/block-indexing) ## Find Hidden Noindex Issues Before Pages Disappear A page can look completely indexable while an unnoticed X-Robots-Tag header quietly removes it from Google. Contact MoxSEO for a technical indexability audit that checks both page-level robots directives and HTTP response headers across the URLs that matter most. We can trace unexpected noindex rules back to server configuration, CDN settings, security tools, WordPress plugins, or path-level directives, then provide a clear recovery plan without removing exclusions that were added intentionally. [Schedule a consultation with MoxSEO](https://moxseo.com/schedule-consultation/) to review missing pages, unexplained traffic drops, conflicting crawler directives, or indexing problems that do not appear in the HTML. We will help you identify the source of the issue, prioritise affected URLs, and restore the correct indexing signals across your website. ## Frequently asked questions ### Does X-Robots-Tag override the meta robots tag? They do not override each other. If either says noindex, the page is not indexed. The most restrictive instruction wins. ### Can I see X-Robots-Tag in my browser? Not in view-source. Open developer tools, go to the Network tab, reload the page, click the document request and read the response headers. ### Why would anyone use the header instead of the meta tag? Because it works on files that have no HTML head. It is the only way to apply noindex to a PDF, an image or any other non-HTML resource. ### How long does it take for a page to come back after removing it? Usually days rather than weeks once the page is recrawled. Requesting indexing in Search Console generally speeds it up.