[mb-datafeed] Live data-feed interruption

[Robert Kaye]

Hi!

As you may or may not have noticed, MusicBrainz is suffering from  
some growing pains that are manifesting themselves in database problems:

http://stats.musicbrainz.org/mrtg/catbus.cgi?log=catbus-loadavg

Our database server is overloaded and Postgres is acting up a bit.  
With some research into matters, Dave Evans upgraded the database  
server to 8.0, which should have better performance and overcome some  
of the strange load growth that we've seen. (our database load is  
rising faster than our overall traffic -- quite curious).

Just before the upgrade, the database server crashed IN THE MIDDLE OF  
CREATING A REPLICATION packet and the packet was lost. Losing one  
packet is bad, since that leaves the database slaves in an  
inconsistent state, and future replication packets will fail to  
apply. The only way to recover from this is to reset the replication  
and export a clean database that all the slaves need to import to  
restart the replication stream.

This means that if you operate a live data-feed replicated  
MusicBrainz server, you will need to dump your current database and  
import this data snapshot in order to get things going again:

ftp://ftp.musicbrainz.org/pub/musicbrainz/data/fullexport/ 
20051231-114240

I realize that re-importing the data is a hassle -- I do apologize  
for this! I'll personally take an action item to create some sort of  
recovery system that allows us to recover lost packets in order to  
prevent service interruptions like this from occurring again.

Fortunately we have just completed an end-of-year fundraiser that  
lines the coffers with $1300 earmarked for purchasing more hardware  
to bring a another server online. This new server will likely be a DB- 
mirror server so that the massive crowds of people wishing to use the  
tagger will not impact our main database server. The main database  
server should be kept exclusively for people wishing to make changes  
to the database.

If you have any questions or comments, please post your questions  
here or mail me directly.

Again, sorry for the hassle!

--

--ruaok      Somewhere in Texas a village is *still* missing its idiot.

Robert Kaye     --     rob at eorbit.net     --    http://mayhem-chaos.net