Using RSS/Atom feeds to discover new URLs

Google uses a number of sources to find new webpages, from the links founded on web to the submitted URLs. Google aims to discover new pages quickly so that users can find new content in Google search results as soon as they are live. Google recently launched a new feature that uses RSS and Atom feeds for discovering the new webpages on World Wide Web.

RSS/Atom feeds have been very popular in recent years as a mechanism for publication of the content. They allow users to check the latest content from publishers. Using feeds for discovery allows Google to get new pages in to its index more quickly than any other traditional crawling method. Google may use many potential sources to access updates from feeds including reader, notification services, or direct crawls of feeds. Google might also explore mechanisms such as PubSubHubBub to identify updated items.

If your RSS/Atom feeds are used by Google for discovery of new webpages, it should first be allowed by the respective robots.txt. To find out if Googlebot can crawl your feeds and find your pages as fast as possible, test your feed URLs with the robots.txt tester in Google Webmaster tools.