A New Server for BookMooch

September 23, 2007

Xeon
BookMooch is now running a new server. I’m hoping this solves the speed problems for a good while.

I emptied out the full text index, so I could rebuild it from scratch, which it is now doing. I think it could take as much 3 days for all the books to re-appear in the full text searching. Searches will return only a partial catalog of books until this reindexing is complete.

For the geeks: the reason I switched to a new server is that after rewriting the database layer myself in C this August, the CPU was no longer what was slowing BookMooch down. Instead, the server was spending 80% of its time waiting for the disk drive. Switching to a faster disk drive (actually, 3 of them) with large caches helped a bit, but not that much.

So, I wanted to move to a 64 bit operating system, so I could tell my hand-written database layer to cache the entire database in memory. Database writes invalidate the cache, and write to disk via a buffer every 60 seconds. When BookMooch starts up, I now do an entire table scan of the user and book database tables, so that they reside in memory. Those two BookMooch tables currently fit inside 3.5gb of RAM, and they’re the most used tables.

The new server is:
– running Debian 4.0, 64 bit version
– has 8gb of RAM
– has one 250gb SATA drive
– has 2 XEON CPUs of 4 cores each (ie, it acts like 8 CPUs)

BookMooch uses aolserver as its web server, is written in Tcl with portions written in C (such as the database layer). Other than ImageMagick to resize people’s uploaded photos, and TclDOM for reading the XML returned by Amazon and OCLC, there are no libraries in BookMooch: everything is custom written for BookMooch itself.

Benchmarks: A really complicated page, such as a full text search for “cards” (which has several thousand database lookups), currently takes 0.16 seconds to render, and apache bench can pull 85 pages per second of that page (ie, the multiple CPUs kick in). A simple page, such as a book details page, takes 0.0025 seconds to render

23 Responses to “A New Server for BookMooch”

  1. Josh said

    Aha, so that’s why the site was down briefly! 🙂

    Also, wow! It’s flying now! Thanks again for all of your hard work on BookMooch, John!

  2. I just found that emails from BookMooch aren’t being delivered.

    They are being queued, though, so as soon as I figure this email server problem out, they should all be delivered.

    Sorry about that (sigh, new servers always cause trouble)

    -john

  3. I’ve fixed the email sending problem. However, all the emails that were queued up in the 1st hour of BookMooch running on the new server have been returned to the sender.

    So, if you got an email from BookMooch that says:
    “Mail delivery failed: returning message to sender”

    you should probably send an email reminder from your pending page of the mooch you just placed.

    New emails won’t have that problem.

    -john

  4. Greg said

    Thanks for all the work John. I think I might wait three days before doing some more Mooching as BookMooch without search is kinda useless.

  5. (sigh) I just discovered that the enter key on the numeric keypad is mapped to “ctrl-c” on this new machine, thereby crashing my BookMooch. I’m running the database cleanup right now, but BM will be down for about 15 mins while this runs, and hopefully nothing serious this time.

    -john

  6. Zillah said

    Thank you so much for all your stirling work making BookMooch a site we call all enjoy, all over the world. My sincere gratitude!

  7. Bill said

    That explains it. It seemed kind of surprising that I didn’t get any search results from “Dune,” “Frank Herbert,” “James Joyce,” or “Ulysses”

    Awesome site though. Woulnd’t have the library I have without it.

  8. Tweena said

    John, you’re a jewel among humans! Not to be over the top, but I just can’t tell you how grateful I am for your creativity and generosity.

    BookMooch has restored my faith in humanity! Thank you for your work.

  9. Heather said

    YAY for new server! That’s wonderful! I’m sure all the little bugs will get straightened out, be sure not to stress about them John.

  10. You are the man, John, thanks!

  11. Martha said

    Thank You. You are Terrific!

  12. Jo said

    Ok, so no change in life is without it’s hiccups, but this new server is still pretty amazing. Although I do have to wonder how long it will take before we all start taking the amazing speed for granted?:-)

    Thanks, John for everything you do for us!

  13. John: Your selfless devotion of time and energy make me want to go out and do good things for humanity… kinda like watching that Liberty Mutual insurance commercial, except in a bookish sort of way. You’re awesome!

  14. neko said

    ummm, did you know that there aren’t ANY books listed? I hope this gets fixed soon. (since it been like this since at least yesterday)

  15. Ashley said

    Yes that is true when you go to browse books it shows 0 books listed in the Bookmooch system.

    Thanks for all your hard work I know it can be a headache. ; )

    Blessings, Ashley

  16. Nanci said

    Sorry for all the headaches it has caused you but you are the Man! I agree. Thanks for making all of this possible for us bibliophiles.
    Nanci

  17. I fixed the problem with the Topics pages, http://bookmooch.com/topics – those pages work correctly now.

    This also fixes a long-standing problem with those pages not refreshing with new books. The big “topics” pages are all cached, and the cache was supposed to be cleared nightly. However, due to a mistake on my part, the cached pages were staying there permanently until I got around to cleaning it out by hand, which was (ahem) infrequently.

    -john

  18. Vanessa said

    All I can say is WOW!!!!!!!! Thank you so much!!!!

  19. Aliera said

    Not to be unappreciative, but how soon will the full text search be finished? My mooching finger is getting itchy.

  20. re: search results missing

    FYI, after 5 days of progress (see msg five days ago), the full text re-index-from-scratch was only at 150,000 book (of 750,000). I had told BM to reindex from scratch, because so many words in the index had gottten “damaged” over the first 14 months of BookMooch’s existence.

    This morning, I stopped the reindex, erased the whole search index (again) and told it to restart reindexing, but this time without the book review/descriptions, as that seems to be about 75% of the effort. As of right now, 9h after I’ve restarted, it’s gone through 160,000 books, with another 600,000 to go. So, in about 2 or 3 more days, the search reindex should be complete (I hope)

    So if you’re having trouble finding books via the full-text-search, that’s why. Wait 72 hours, and all should be well.

    -john

  21. Andrea said

    Anything new on the updated server front? I’m still finding that books from my inventory, more often than not, do not appear when typed into the search box.

  22. re: searching

    As of right now, the reindex has done 610,000 books, with 140,000 more remaining to do. Another 2 days, I think, and the search index should be up to date.

    -john

  23. Evan said

    Is this a related issue?

    I’ve noticed that when I’m viewing a book by Jim Butcher, when I click on his name BM doesn’t show any books by him. But I know BM has books by him since I have them in my Wishlist.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: