Last week, Yahoo announced that they now support the X-Robots-Tag in the HTTP header.

What is the X-Robots-Tag?

Google introduced the new X-Robots-Tag directive in 2007 to allow webmasters to control access to non-webpage documents, such as Adobe PDF files, videos, and audio files.

The X-Robots-Tag is included in the HTTP header of a document. The HTTP header is the initial reply of a server to a query. It contains information about the document that follows, including its content type, creation date, character set, and encodings, among other details.

The new X-Robots-Tag allows webmasters to add information about search engine indexing to the HTTP header.

Which commands are supported by the X-Robots-Tag?

At this time, the X-Robots-Tag supports the following commands:

~ X-Robots-Tag: NOINDEX (Use this tag if you don’t want to show the URL in Google’s search results)

~ X-Robots-Tag: NOARCHIVE (Use this tag if you don’t want to see a Cache link in the search result pages for the document)

~ X-Robots-Tag: NOSNIPPET (Use this tag if you don’t want to display a summary in the search result pages.)

~ X-Robots-Tag: NOFOLLOW (Use this tag if you don’t wish for Google and Yahoo to index the links in the page.)

Google and Yahoo currently support the X-Robots tag. Other search engines don’t support the tag yet.

How to add the X-Robots-Tag to your documents

It depends on your server and the method you use to create your web pages to add additional tags to the HTTP header. If your web server uses Apache, you can use the .htaccess file to modify your HTTP headers.

In general, you shouldn’t modify the HTTP header unless you’re 100% sure what you’re doing. A broken HTTP header can keep search engines away, so that no search engine will index your website.

Instead of using the X-Robots-Tag, you can also add meta tags with the same functionality to your web pages:

You can edit these tags directly in the < head> section of your web pages. They have the same effect as the corresponding X-Robots-Tags explained above.

Do you need the X-Robots-Tag or the corresponding meta tags?

If you don’t want to limit access to your documents or if you only have regular web pages on your website, then you don’t need the X-Robots-Tag or meta tags.

The X-Robots-Tag has been designed to restrict access to your documents. That means it doesn’t help increase search engine rankings.

The new X-Robots-Tag and the meta robots tag allow you to restrict access to your documents.
To ensure that search engines index all your web pages, you need to optimize them.

Search engine robots must be able to parse your web pages easily, and they must be able to find the correct elements on your web pages. If you optimize the proper web page elements with relevant keywords, your web page will rank higher in search engines for those keywords.