Google Adds SiteMaps in robots.txt of All Blogger Blogs
Google is continuously improving the Blogger platform - their latest feature discussed below may help get your blog content on search engines more quickly.
For non-geeks: Sitemap files enable you to let search engines know about the new or updated content on your blog that should be indexed by their crawlers. Earlier you could point Google to your Blog's SiteMap file through the Google Webmasters control panel but that's no longer required.
Now whenever your publish a new blog post on Blogger or update some previous post, that information will automatically become available to Google and other search engines - you don't have to do anything at your end. How cool is that.
For geeks: Google has now added the Sitemap: directive to the robots.txt of all blogspot blogs which actually points to the default xml feed of that blog containing the 25 most recently updated blog posts. Here's an example:
User-agent: *
Disallow: /search
Sitemap: http://abc.blogspot.com/feeds/posts/default?orderby=updated
This is an excellent development because the Sitemap: sytax is supported by all major search engines including Google, Yahoo! and Windows Live - their search crawlers can therefore auto-discover your new or updated blog content based on the sitemap file.
Related: Verify Google Sitemaps for Blogspot blogs
If you are getting Invalid XML namespace errors and warnings with Blogspot sitemaps file, read solution here
Subscribe in a reader
Download Free Google Software:
Reader Comments:
This is cool; I think I've finally got the correct sitemap in Google Webmaster tools associated with my Blogspot blog again.
But I wish there were a way for Blogger to recognize more than 26 URLs submitted in Google Webmaster tools.
For example, my new site, WatchFreeEpisodes.com, on WP platform already has 80 or so URLs recognized thru the sitemap I uploaded to my server -- something I wish we could do fully on Blogger.
Being able to recognize more than 26 urls submitted seems so much better.
Thanks,
PaulaNealMooney.com
Paula - why should that be a concern ? Googlebots index sites pretty frequently so they'll have the necessary information before you publish the next set of articles.
Amit,
But there seems to a issue that after this new feature i get errors in my webmasters Tools.
I get errors under "URL restricted by robots.txt" and all these errors are for search labels..
Any idea why this is occuring...
Cool. But since I have burnt my feed at Feedburner, do I have to do anything?
I'm a non geek.
Excellent ! But since when did the Sitemap: attribute get added to the robots.txt protocol ?
@kishore - Google has disallowed indexing of label with the new robots.txt - see the "Disallow" line in the above screenshot.
@ranjan - You're lucky - you don't have to do anything at your end. Even if you are using Feedburner, the sitemap file would still work.
@Anjanesh - I think that was last year when Google, Microsoft and Yahoo! jointly agreed to support the Sitemap: syntax in robots.txt.
hi,
check out http://fundubytes.blogspot.com/robots.txt which is blocking
web crawlers to crawl even though "Add your Blog to our listings?"
under basic settings is set to yes and blog template contains meta tags for enabling robots crawl
even,
Sitemap: http://labnol.blogspot.com/feeds/posts/default?orderby=updated
is missing.
please suggest me what to do!
If I go to the Site Diagnostics in my AdSense account, under "Blocked URL", it has the URL for a lot of my posts, and under the "Reason Blocked", it says "Robots.txt File".
What does that mean, and how can I fix it? Indeed, none of those pages got index, because if I do a Google search in quotes with the exact title of the post, I get no results.
There's a small problem with this change.
It breaks Adsense. Both Blogger sites I administer (Mine and a friends) are now blocked by adsense. The reason given is "robots.txt"
Why Google's own Blogger would implement a change that broke adsense is beyond me, but now I have to look into alternate advertising sources for the blogs.
Have a question? Need help? Visit the forums ».