It’s a website owner’s worst nightmare. Having a site removed from Google’s index means you no longer receive traffic from the search engine. With no search engine traffic, your sales plummet, and it can greatly affect your online business’ future. Being removed from Google’s index isn’t permanent as long as you fix the problem. The problem can be egregious violations of Google’s guidelines, or you could have made a simple server configuration error. Here are some quick ways to identify the issue and get your site back into Google’s good graces.
Google Webmaster Tools URL Removal
One of the most common reasons for a site to drop from the index is misusing the URL removal tool in Webmaster Tools. Webmaster Tools has a section where webmasters can remove URLs from the index quickly. It’s intended for pages that have sensitive information that you need removed, or it’s for times when you have several pages indexed that you no longer want in the index.
Many SEOs and webmasters misuse this tool to remove pages that return a 404 (temporarily not found) server message or for canonical issues. The tool should never be used for either one of these issues. The result is the SEO or webmaster accidentally removes the entire site.
In some cases, disgruntled employees or SEOs given access to your Webmaster Tools console will maliciously remove the domain from the index. For this reason, you always want to give SEOs and employees “read only” access to Webmaster Tools.
To remedy this issue, go into Webmaster Tools and click the “Remove URLs” link in the “Google Index” section. Click “All” in the drop-down to view all removal requests. Click “Cancel” if your domain is listed. It takes a few days for the URL to return to the index, but Google will return the site to its previously indexed status.
Spun Content or Extremely Poor Quality Content
Spun content makes horrible content, and you should never engage in machine-created content. Software called a “spinner” takes specific adjectives and adverbs and attempts to create unique content by replacing certain words with related words. The result is horrible content that’s never useful for readers.
If your site is caught spinning content, Google places a manual penalty on the site, and in severe cases, Google removes the site entirely from the index. The only way to recover from this issue is to remove all spun content from the site. In some cases, this is all the content on the site. However, you must remove all spun content to have the manual penalty removed. After you remove the content, file a reconsideration request in Webmaster Tools.
If your site has any type of manual penalty, Google recently started displaying manual penalties in Google Webmaster Tools. Click “Search Traffic” in the navigation panel and click “Manual Actions.” This page tells you if any penalties are applied to the site, so you know you need to take action to fix indexing issues.
Server Configuration Errors
Any type of server configuration error that blocks Google from accessing the site affects your site’s index status. If you have DNS errors, it means Google can’t access the domain. If your server times out, it means Google can’t crawl your site.
One big issue with websites and indexing is poorly configured firewalls. Some web hosts configure firewalls to block too many consecutive page requests. Several requests made consecutively can trigger denial of service (DOS) detection software.
While this is good when it’s a malicious hacker, bots crawl website pages very fast and the crawling can look like a DOS attempt. If your host is incorrectly detecting a bot as a DOS attempt, the firewall blocks Google and your site can drop in rank or be entirely removed. The only way to correct this issue is to ask your host to fix the firewall configurations or move host providers entirely.
If any server configuration errors exist, Google usually reports these errors in Webmaster Tools. Check Webmaster Tools for any errors that could affect your site’s index such as 500 errors or DNS errors. Server 500 errors are general errors, but they are typically coding issues. DNS errors mean Google can’t resolve your site’s IP with the domain name. Your website host can help you fix any DNS errors.
“Noindex” on Your Pages
The “noindex” meta tag tells Google not to index your pages. The meta tag can be manually coded into pages, or some content management software such as WordPress and Joomla have configurations to set the tag in the software’s control panel.
To find out if you have the meta tag set in your site’s code, open your website in any browser. Right-click the page and select “View Source Code.” The meta tag is within the “head” HTML tags. Find all meta tags in this HTML section and make sure none of them have “noindex” in the code.
To fix this issue, remove the meta tag and wait for Google to recrawl the pages. If you use a content management system such as WordPress or Joomla, check the software’s control panel for any settings that block search engines from indexing the pages.
The Site is Brand New and You Need Patience
Site owners think that Google indexes instantly, but Google needs to crawl the site before indexing any pages. Indexing using takes a few days unless you have a backlink to the site. If you submit a sitemap in Webmaster Tools, wait about two weeks before panicking. In most cases, your site is indexed within a week, but some sites take longer.
These are just a few ideas when checking for indexing issues in Google. Google rarely indexes all pages in a domain, but as long as your site focuses on users and you maintain quality, Google will index and rank your website in time. While indexing is easy, ranking is difficult, so always focus on users and quality and have patience when working with your site’s search engine optimization.