[mb-users] Why is robots.txt disallowing all of this?

[Steve Wyles]

On Sun, 4 May 2008, Philip Jägenstedt wrote:

> I would suggest appending the following to the current robots.txt:
>
> User-Agent: Googlebot
> Allow: /artist/
> Allow: /release/
> Allow: /track/
> Allow: /show/ (for seeing full AR listings)
>

IMO that would be a little silly. From what I recall, MB pays a specific 
amount for the bandwidth used at Digital West depending on the average 
bandwidth used in the last month. Anything that increases hosting cost and 
adds additional load to the servers needs detailed investigation.

Steve