[mb-users] Why is robots.txt disallowing all of this?

[Robert Kaye]

On May 3, 2008, at 12:21 PM, Philip Jägenstedt wrote:

> Wondering why Google doesn't index MusicBrainz very well I turned to
> http://musicbrainz.org/robots.txt
>
> It seems most of everything is off limits to the search engines. Why?
> Wouldn't search traffic generate more traffic to MusicBrainz, giving
> us more new users?

We do not support HTML caching right now.

If we can support caching, then having the google bot come by becomes  
less of an issue. And we can't support caching until we have last- 
changed-on timestamps for our data entities. Once we have those we can  
have all sorts of goodies like RSS feeds for entities, what portions  
of my music collection in MB have recently changed and HTML caching.

Until then, no go on the google bot.

--

--ruaok      Somewhere in Texas a village is *still* missing its idiot.

Robert Kaye     --     rob at eorbit.net     --    http://mayhem-chaos.net