Guru Site HealthIndexing
Discourage search engines from indexing this site — the checkbox that hides your whole site.
It is one checkbox, three clicks deep in Settings, and it is ticked by default on a great many new installs. If it is still ticked when you launch, Google will not index you — and nothing on your dashboard will be red about it.
What the setting actually does
In Settings → Reading there is a checkbox labelled Discourage search engines from indexing this site. When it is on, WordPress does two things:
- It adds
<meta name="robots" content="noindex, nofollow">to every page of your site. - It serves a
robots.txtcontainingDisallow: /.
Those are not subtle hints. The meta tag tells a crawler not to put the page in the index and not to
follow any link on it. The robots.txt rule tells it not to request the page at all. Together
they mean a search engine that obeys the rules will not list you, and the ones that obey the rules are
precisely the ones you want.
The label is softer than the behaviour. People read “discourage” and assume it nudges rankings down a little. It does not. It is a request for total exclusion, and Google honours it.
Why it is on when you did not turn it on
Almost nobody ticks this box on purpose and then forgets. It gets switched on in four ordinary ways:
- The install asked. Several one-click installers at hosts — and WordPress’s own famous five-minute install — offer a “this site is under construction” option that sets it.
- A staging site became the live site. Staging environments set it deliberately, correctly, so the half-finished copy does not compete with the real one. Then the staging copy gets pushed live, and the setting rides along.
- A developer set it during the build and handed the site over without unsetting it. This is the most common one, and the most expensive, because the handover is also the moment everyone stops looking.
- A migration plugin copied it. It is stored in the
blog_publicoption, which is an ordinary row inwp_options, so every full-database migration carries it across.
How to tell whether it is on, in ten seconds
Three ways, in order of how little you have to trust:
1. Look at the setting
Go to Settings → Reading and look at Search engine visibility. If the box is ticked, that is your answer and you can stop reading this section.
2. Look at what you are actually serving
Visit yoursite.com/robots.txt in a browser. If you see this:
User-agent: *
Disallow: /…then you are telling every crawler to go away. A healthy file either allows everything or disallows a few specific paths.
3. Look at the page source
Open your home page, view source, and search for noindex. If you find
content="noindex, nofollow" in a robots meta tag, the page is excluded. This is the check
that matters most, because an SEO plugin can also add noindex independently of this
setting — so the box can be unticked and the tag still be there.
How to fix it
Untick the box in Settings → Reading and press Save Changes. That is the whole fix.
The meta tag disappears on the next page load and robots.txt reverts immediately.
If you would rather not hunt for it, Guru Site Health flags this as a finding and offers a one-click fix for it on the dashboard, along with an Undo if you ticked it on purpose. It is the first thing the scan looks at, because it is the finding with the worst ratio of “trivial to fix” to “catastrophic to miss”.
The part people skip: unticking it does not undo the damage
This is where most advice stops, and it is the half that costs you months.
Unticking the box lets crawlers back in. It does not summon them. If the setting was on for a while, Google has already tried, been refused, and slowed down how often it bothers. Left alone, a small site can sit there for weeks before anyone comes back to look.
So after you untick it, do these four things in this order:
- Confirm the tag is gone. Reload your home page and check the source again. If
noindexis still there, it is coming from your SEO plugin, not this setting — check Yoast or Rank Math’s own visibility options. - Check your permalinks are not plain. If your URLs look like
?p=123, fix that before you ask to be indexed, because the URLs you get indexed are the ones you will be stuck with. - Make sure you have a sitemap. WordPress has had one built in since 5.5 at
/wp-sitemap.xml, unless something switched it off. - Tell Google directly. Verify the site in Google Search Console, submit the sitemap, and use URL Inspection → Request Indexing on your most important pages.
That last step is the one that turns weeks into days. Getting a WordPress site indexed by Google walks through it properly.
How long until you are back
There is no promise anyone can honestly make here, but the pattern is consistent. A site that was blocked for a few days and is then unblocked and submitted through Search Console is usually back within a few days. A site that was blocked for a year is starting from a standing start, and competing against pages that have had that year to accumulate links. The setting is cheap to fix and expensive to have left on, and the cost scales with how long nobody noticed.
Then leaving it ticked is the right call — that is what it is for. The failure is not having it on; it is having it on without knowing. Put a reminder on the launch date, or install something that tells you.
Related settings that hide pages the same way
The checkbox is the loudest version of this problem, but it is not the only one. If you are not being indexed and the box is unticked, check these:
| Where | What to look for |
|---|---|
| SEO plugin settings | A per-post-type or per-taxonomy “noindex” toggle |
| Individual posts | A per-post “noindex” switch set by someone else |
A custom robots.txt | A real file on disk overrides the one WordPress generates |
| Your host | Staging environments often block crawlers at the server, above WordPress |
| Password protection | A site behind HTTP auth cannot be crawled at all |
X-Robots-Tag | An HTTP header that does the same job as the meta tag, invisibly |
The header one catches people out badly, because nothing in WordPress shows it. You can only see it by
looking at the response headers — in your browser’s Network tab, select the document request
and read the response headers for X-Robots-Tag.
Frequently asked questions
Does “discourage search engines” completely block Google?
In practice, yes. It adds a noindex, nofollow robots meta tag to every page and serves a robots.txt with Disallow: /. Google honours both. WordPress uses the word “discourage” because it cannot technically force a badly behaved crawler to comply, but every search engine you care about complies.
How do I know if the setting is on without logging in?
Open yoursite.com/robots.txt. If it contains Disallow: / under User-agent: *, crawling is blocked. You can also view the page source of your home page and search for noindex.
I unticked it and I am still not in Google. Why?
Three likely reasons. First, unticking lets crawlers in but does not bring them — submit your sitemap in Google Search Console and request indexing. Second, an SEO plugin may be adding its own noindex tag independently. Third, an X-Robots-Tag HTTP header set by your host or a staging environment does the same job and is invisible from inside WordPress.
How long does it take to get indexed after I fix it?
It depends mostly on how long the block was in place. A site blocked for days and then submitted through Search Console is often back within days. A site blocked for a year is effectively starting fresh. Nobody can give you a guaranteed number, and anyone who does is guessing.
Should I use this setting for a staging site?
Yes — that is exactly what it is for. The mistake is not switching it on, it is pushing staging to production with it still on. Better still, block crawlers at the server level on staging, so the setting never travels with a database copy.
Where is this setting stored in the database?
In the blog_public option in wp_options. 1 means visible, 0 means discouraged. Because it is an ordinary option row, any full-database migration or clone carries it from one environment to another.
Find these on your own site in about ten seconds
Guru Site Health runs 32 checks across updates, security, database, media, accessibility and SEO, and gives you one score. Thirteen of the findings can be fixed from the dashboard in one click, and every fix can be undone. It makes no external requests — nothing about your site ever leaves it.