Alas, some data loss

October 25, 2007

Sadface
I upgraded the BookMooch server today, from 8gb of RAM to 24gb.

This is important because the server was so overloaded that the daily backup, started at 2am, was taking until 3pm to finish (13 hours!).

I just did a full backup now, with the server running, and it took 11 minutes.

Alas, there was some trouble with the server upgrade, and a bit of data was lost. I had a backup from yesterday afternoon, which was used to restore some of the data.

What this means is that for some of you, you may be missing some mooches that you did between yesterday afternoon and today.

If you can figure out what to do on your own (such as re-contacting the person you were mooching with), that would really help us out. If you need help, send an email to tech support and they’ll help you out.

Yes, I know the data loss problem is really awful, believe me, I really feel it. I know what I need to do to fix it, and it’s soon on my todo list, but unfortunately to make the data loss problem go away, BookMooch will have to do more work, and hence be a bit slower, so I need to both proceed carefully, and also make sure that there’s plenty of speed to keep up with daily demand.

19 Responses to “Alas, some data loss”

  1. Taneli T said

    Just to clarify, when you say “I know what I need to do to fix it, and it’s soon on my todo list”, do you mean this particular data loss case that just happened or something you’re going to do to prevent potential data loss cases in the future?

    I mean, should I add those books that disappeared from my inventory myself (which I think you are saying), or wait for them to appear there automatically as part of this coming fix?

    24 gigs sounds awesome.

  2. Mark W posted these tips for dealing with the data loss. Note that there won’t be any further recovery from BookMooch or me — you should follow Mark’s advice if you can.

    -john

    The data loss seems fairly widespread, and appears to cover 24-30 hours previous to the outage earlier today.

    So I thought it would help to post the following tips:

    — Luckily Bookmooch sends notice of all requests placed and received, so hopefully many will be able to reconstruct yesterday’s activity. If you deleted the e-mails they will still exists if you have a ‘trash’ folder for discarded e-mails. If not, in many cases, the member on the other end of the transaction may well be in conatct with you.

    — If you added any new books (or added comments), each will have to be re-added (or removed any for that matter, re-remove them).

    — Same for wish-list adds, or save-for-later adds, friends added, charity or smooches given and received.

    — If you marked books as received, you will need to back-track and check them in once again.

    — If you marked books as sent, the same.

    — If others received your books, they will not show up on your feedback. But If the other member also lost data (not sure how widespread the data loss was yet), then he/she will hopefully check the book in again.

    — To deal with any requests made of your account: If you have the notification e-mail, you should be able to contact this person (or if you don’t have the e-mail, he/she might contact you) to reconstruct the transaction.

    If wishlisted, it may be best to ‘hand-enter’ the book to avoid confusion, rather than just directly re-enter it. Either way, please honor the initial request placed. Also, if anyone gets a rejection due to the error, please be understanding and do not leave negative feedback for it (probably best just to cancel the request once you find out the book had been mooched and the data lost).

    — Likewise, you should be able to contact those from whom you mooched and ask that the book be re-listed so the mooch can be placed again.

    If the person shows the mooch but you do not, more steps are necessary, contact tech support if this has happened with your account (but it appears that most were affected, so this might not come up).

    Dependent on your activities, you may have suffered an overall point loss, if you are not able to straighten things out within a few days, let us know and your points can be adjusted.

  3. In most cases, you shouldn’t have lost any points, rather your account should have simply reverted to its state yesterday. If you have less points now than right before the problem, it should be simply because the mooches you received, that were lost in the data-loss, have been lost.

    That being said, there are some cases where some points were lost, and if that’s the case, please send a message to tech support explaining *briefly* (they’re getting a lot of email, make it easy for them, please) what the problem is and how many points you need to correct it.

    -john

  4. DO NOT post your tech support requests on this blog — the tech support volunteers do not reply on this blog.

    If you need tech support, please send an email to the BookMooch tech support volunteers at this url:

    http://bookmooch.com/m/email_support

  5. Mark Williams said

    One addendum: Those who signed up during the outage will have to do so again, so please allow them time to do so.

    If you had a transaction with such a new user, you will likely see a ‘no such user’ message if you try and look them up.

    It is possible some may not sign up a second time, but I think most will, hopefully.

  6. Mark Williams said

    Update: After going through a few dozen accounts, the data loss goes back a day earlier for many, so begins around 3-6 A.M. (U.S. EST) on Oct 23 and extends until the beginning of the outage today, Oct 25 at aprox. 10:00 A.M. (U.S. EST).

    Mark W

  7. Heather said

    *sends hugs to John and Mark and all staff*

    I griped a bit in the forums, but I was just a bit shell-shocked, yunno? I really do appreciate everything you all do for this site, and I can’t even IMAGINE what this data loss has done to you all, stress-wise. I hope it all calms down soon, for everyone’s sake.

  8. William said

    I was wondering why I lost about 4 points – unfortunately I put about 40 more books online the other evening and spent a whole lot of time describing them. Oh well. I’ll put them back in inventory.

    I wonder if some of the strain on the server is some of the unfixed bugs which also “bug” me… Points page and Wishlist page. These are especially bad – over the last year whenever I click on “Points” my browser feezes up for about 15 minutes until it loads up all the thousands and thousands lines of html. I usually force quit and forget it – I just can’t view anything other than my most recent point history. But if I really really need to see something that happened earlier, I let my browser sit there for 15 to 17 minutes until it loads up the point history. That must be doing something bad on the BookMooch server end too. Same thing with Wishlist – when I click on Wishlist, my browser freezes for 15 minutes or so until all that data is loaded on the 1500 or so books. Things aren’t divided into edible information chunks. I hope this bug is going to be fixed to make it healthier on the BookMooch server as well as all our computers. Can’t Points history be divided up into smaller edible chunks like most everything else except Wishlist? Everytime BookMooch has crashed over the last year I’ve always wondered to myself if too many people were doing something too user-intensive at the same time, like checking on their Points history.

  9. Mark Williams said

    I’ve re-written my recovery tips to better reflect the current sitauation:

    The period of loss varies, but for most extends from aprox. 3-6 A.M (U.S. EST) on the morning of Oct 23 until aprox. 10:00 A.M EST Oct 25.

    If you are sure you lost data from before this outage period, please let us know, as there will be a few extra steps to take.

    — Luckily Bookmooch sends notice of all requests placed and received, so hopefully many will be able to reconstruct the last 2 days of activity. If you deleted the e-mails they will still exists if you have a ‘trash’ folder for discarded e-mails. If not, in many cases, the member on the other end of the transaction may well be in contact with you.

    — If you added any new books (or added comments), each will have to be re-added (or removed any for that matter, re-remove them).

    — Same for wish-list adds, or save-for-later adds, friends added, charity or smooches given and received.

    — If you marked books as received, that data appears to be lost, but the books should also no longer show as pending (if they do, please mark them as received).

    — If you checked in more than a couple and cannot do so again, add up any point loss from the 1/10 bonuses, and let us know (in a few days, please, see below) and your points can be adjusted.

    — If you marked books as sent, accepted, or delayed, check your pending records to see if this needs to be done again.

    — If others received your books, they will likely not show up on your feedback. Sorry for the loss of feedback!

    — To deal with any requests made of your account: If you have the notification e-mail, you should be able to contact this person (or if you don’t have the e-mail, he/she might contact you) to reconstruct the transaction. To do so, simply respond back to the email.

    If the book had been in your inventory prior to the data loss then it can be re-mooched directly, if not, it must be re-entered:

    If wishlisted, it may be best to ‘hand-enter’ the book to avoid confusion, rather than just directly re-enter it. Either way, please honor the initial request placed. Also, if anyone gets a rejection due to the error, please be understanding and do not leave negative feedback for it (probably best just to cancel the request once you find out the book had been mooched and the data lost). It may also be a good idea to add a ‘hold for xxx’ note in the condition field.

    — Likewise, you should be able to contact those from whom you mooched and ask that the book be re-listed so the mooch can be placed again. (For mooches you placed, only the ‘accepted’ or ‘delayed’ notices have the sender’s e-mail, the ‘sent request’ notice does not, for these: click on the bio link in the email (right below the book details link), from here you can send an e-mail with the re-mooch request. It would be helpful to cut and paste a link of the book in question). If someone else appears to have mooched your book, please contact the sender and ask it be held for you.

    — If the other person was a brand new user, they may have to sign up again, and will show as ‘no user found’ until they do. If you find this message from looking up mooches/requests, then you will not be able to e-mail the new member if you only have a ‘sent request’ notice, but will be able to do so if you have any other notices, which should include his/her e-mail in the ‘sent from’ field. (if you had any extra contact with such a new user, outside of the official notices, please let them know of the outage).

    — Once books are re-entered, or otherwise made available again, it would be helpful to email the moocher/sender with a link back to the detail page of the book (cut and paste). Or just reply directly to email if it includes the other member’s e-mail in the ‘sent from’ field.

    — If you have deleted you e-mails form the past two days, you can try looking up the book in question. You may be able to find it if was not newly entered, or has been re-entered. Oddly enough, the book (if the same exact edition) should show the last sender and moocher of the book, even if this was in the period of data loss. So you may be able to reconstruct the mooch in this way.

    — Dependent on your activities, you may have suffered an overall point loss, and if you are not able to straighten things out within a few days, let us know and your points can be adjusted. But please give this a little time as we would rather not make a series of adjustments, and it may take up to a week for all the lost transactions to be recreated.

    Sorry for all the trouble folks, hopefully it won’t take too long to set things right!

  10. William said

    Well besides all the added inventory, I lost data on some mooched books that came in – they totally vanished and there seems no way to recover the data. They are no longer in Pending, never got put in my History, despite feedback after receiving them, and I can’t find them in BookMooch general book data either, since they were the only copies on BM. If I knew who I got them from I could check if they were in the other person’s history. ALl my trashed emails delete after 2 days, and my trash pickup was this morning, so the envelopes they came in are gone too! Alas, one of those mysteries in life. πŸ™‚ (I do wish the Points loading history would get fixed, though, so I could study my points history. And every time I click on a mooch to examine it then go back to Points History it takes another 15 minutes to reload!)

  11. Taneli T said

    There’s an upside to this server upgrade:

    All the searches I’ve tried happen now instantly, with practically no delay at all.

    Thanks.

  12. moonplant_lady (Bertie) said

    Bookmooch has the best tech support team!
    I was completely amazed that Dave, BookMooch Team member, responded so quickly to my S.O.S.
    so I could let receiving moochers know quickly about the “pending” and sent problem in our accounts.

    A huge “THANK YOU” from PA.

  13. Taneli T said

    To not confuse those new users who joined in the past couple of days and now can’t find their account when they log back in, maybe you could add a note of what has happened to login page, so they would know to create their account again.

  14. John Hopper said

    I was a bit alarmed when I saw his email this morning. On checking my account however, I seem to have suffered very little data loss. Books I inventoried and wishlisted over the last 2-3 days are still there, and those I received and sent are no longer in my pending. The only loss seems to be one feedback point from a book I sent out which was acknowledged as received on 24 August.

    I wonder, is this because I am UK-based? Not sure why that would make a difference, though….

    John

  15. Mark Williams said

    Another tip for those without e-mail records of transactions:

    The quickest way to find a recent record of lost mooches is to look at the detailed view of the exact edition of the book in question, for some reason the last sender and moocher shows the missing data, not to be found elsewhere on the site.

    So, if the book remains in your inventory, just look at the detailed view. If it was added during the outage, and so no longer in inventory, by either you or another, you should be able to do an advanced search and select ‘search all books’ from the
    ‘search wishlists’ pull down. Or just do a standard search if it was added earlier by another.

    This does not help if the book was added by a member who signed up during the period of data loss, but does seem to work otherwise.

    I’ve tried this 20+ times and it has worked in every instance so far.

  16. Maggie said

    I’d say John and the entire tech team have earned a day (or maybe a week or a month) off! Each and every moocher has really gotten way more than fair value for the membership fee. Even with success loading up the site, this is a really great service to book lovers.

  17. Emily said

    John (and team), thank you for working to fix the problem so quickly.

    I lost a bit of my data, but luckily I was able to remember what I’d done that day and re-contact the other members. We worked it out just fine. πŸ™‚

    And I just want to say again how VERY VERY MUCH I love bookmooch. πŸ™‚

  18. Jennie said

    I love all of you and all except for one of mooches have been redone. I am so thankful that Google invented Gmail though!

  19. Mark Williams said

    Thank you all for all the thanks πŸ™‚

    Two requests from the admin team:

    1.) Please do try and reconstruct your mooches placed on Tuesday and Wednesday folks, rather than just requesting points for lost mooches. It may take a few days to hear back from your queries, but it is much better for all to have a proper record of your books having been sent. The admin team is not simply re-adding points for lost mooches, but we can do this after looking at your account in another week or so if you are unable to contact the moocher or sender, but please, please do make the attempt on your own first.

    2.) Feedback for books you had received by moochers, or those you’ve received yourself are gone, sorry about this, but please do not try and reconstruct these transactions, as the books have arrived and it is not necessary (but indeed will cause other problems). If you are new and really want to let people know of your lost feedback, then add a note on your bio message, which all can read. We can make fractional point adjustments for 1/10 point bonuses which have been lost, but please just give us a few day before reporting your loss unless you are in dire need.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: