[mb-users] Why is robots.txt disallowing all of this?

[Philip Jägenstedt]

Sounds reasonable to me, even though I'm not sure there would be a
flood of new users overwhealming us. My guess is that even if we open
all the the things I suggested Google would only index a small part of
it, as MusicBrainz shouldn't have *that* many incoming links (although
Wikipedia and Last.fm have a high pagerank I guess).

The turnaround time for Google isn't that long, maybe a few months, so
it would be possible to close up again if server load or new users
rise too sharply.

What say you?

Philip

On 5/4/08, Kuno Woudt <kuno at frob.nl> wrote:
> On Sat, May 03, 2008 at 12:21:11PM +0200, Philip Jägenstedt wrote:
>  > Wondering why Google doesn't index MusicBrainz very well I turned to
>  > http://musicbrainz.org/robots.txt
>  >
>  > It seems most of everything is off limits to the search engines. Why?
>  > Wouldn't search traffic generate more traffic to MusicBrainz, giving
>  > us more new users?
>
>
> We don't want too many new users at once, obviously anything like this
>  should be considered carefully with regards to load of the servers,
>  etc..
>
>  But there is also the Eternal September [1] kind of thing to consider, if
>  it all possible I would be in favour of very slowly allowing more people
>  to find the site.  E.g. by starting out with allowing google to index
>  only a few albums or artists, and slowly increasing that -- to make sure
>  not only the load to the servers is managed, but also the impact large
>  groups of new users will have on our editor community and the quality of
>  unvoted add release and other edits.
>
>  -- kuno / warp.
>
>  [1] http://en.wikipedia.org/wiki/Eternal_September
>
>
>
>  _______________________________________________
>  MusicBrainz-users mailing list
>  MusicBrainz-users at lists.musicbrainz.org
>  http://lists.musicbrainz.org/mailman/listinfo/musicbrainz-users
>