After denying search engines from indexing the millions of bookmarks stored on its servers for so long, del.icio.us has finally lifted the entry barrier but selectively.
According to the robots.txt of del.icio.us, only spiders from Yahoo!, Google and Ask search engines are allowed to crawl and index the content of del.icio.us - all other bots are banned including msnbot of the Microsoft Live search engine.
User-agent: *
Disallow: /
User-agent: Slurp [Yahoo]
Allow: /
User-agent: Googlebot
Allow: /
User-agent: Teoma [Ask.com]
Allow: /
del.icio.us has also removed the meta robots (noarchive,nofollow,noindex) restriction from all their webpages that earlier prevented search engines from indexing or caching delicious web pages.
As a result, Google has 6 million pages from del.icio.us in their index now while this number was around a million sometime back when the ban was in place.
What does this mean for sites / URLs that are popular on del.icio.us - you could see a surge in search engine traffic but getting listed on del.icio.us won't add any link authority or google juice since all links on delicious are marked with nofollow, very similar to Wikipedia.
Find this article at: http://labnol.blogspot.com/2007/08/delicious-opens-flood-gates-but-only.html
web: http://www.labnol.org/ email: amit@labnol.org
Reader Comments
That's interesting news..I never knew that Google does not index del.icio.us links in the first place..By the way, does it also carry Google juice? Since that would mean people bookmarking their own blogs and websites with multiple fake accounts..
Written on 9/8/07 10:37 PM
What is more surprising is that yahoo had bought del.icio.us, then why was yahoo slurp not allowing to index the site !! any thoughts ?
Written on 10/8/07 12:11 AM
Hi Anand,
Nope, no Link Juice. This is mentioned in the end of the article:
"getting listed on del.icio.us won't add any link authority or google juice since all links on delicious are marked with nofollow".
Written on 10/8/07 2:26 AM