RSS and Atom Feeds - Additional Discovery Method of Google

In the initial years of its launch, Google could discover the existence of web pages only through links. Googlebot could not detect any page that did not have a link attached to it. Over years, the company devised a way to submit individual pages but was not a very successful option for those with large sites. Then Google launched XML Sitemaps in 2005 which was a much easier way for site owners to let Google know about their pages which Googlebot may otherwise miss. But today, there is a much better way by which the company can discover pages using RSS and Atom feeds and this article can be found on Google Webmaster Central blog.

Google always has its eyes out for new content as freshness is the key component for most users using search. Convention system does not always work as Google cannot detect pages which do not have links and these pages become invaluable to the site. The blog post that appeared recently discusses about using RSS and Atom feeds as Sitemaps even if the website owner has not submitted his pages through the Sitemap system. Other feed submission systems are also being scanned like Google Reader and ping services.

What the blog post does not say is if the feeds are being used solely for discovery of pages or if the content obtained from the feeds are used in place of crawling as well. Though the post is titled ‘discovery,’ it says that Google is able to get these pages into the index faster than using traditional crawling methods. Google Blogsearch had initially crawled feed content instead of actual webpages which caused partial indexing. If Google is to use the feed as a discovery method, then it must not be blocked by robots.txt

Popularity: 2% [?]