[mb-users] Why is robots.txt disallowing all of this?

[Kuno Woudt]

On Sat, May 03, 2008 at 12:21:11PM +0200, Philip Jägenstedt wrote:
> Wondering why Google doesn't index MusicBrainz very well I turned to
> http://musicbrainz.org/robots.txt
> 
> It seems most of everything is off limits to the search engines. Why?
> Wouldn't search traffic generate more traffic to MusicBrainz, giving
> us more new users?

We don't want too many new users at once, obviously anything like this
should be considered carefully with regards to load of the servers,
etc..

But there is also the Eternal September [1] kind of thing to consider, if
it all possible I would be in favour of very slowly allowing more people
to find the site.  E.g. by starting out with allowing google to index
only a few albums or artists, and slowly increasing that -- to make sure
not only the load to the servers is managed, but also the impact large
groups of new users will have on our editor community and the quality of
unvoted add release and other edits.

-- kuno / warp.

[1] http://en.wikipedia.org/wiki/Eternal_September