Fast again

For the past month, I’ve been working on making the BookMooch web site fast again. I think I’ve finally licked it.

I keep a daily chart of how many seconds it takes to load up the home page. Here is what the chart looks like over a year and a half:

You can see that the speeds were quite good from October 2008 until August 2009 when they started to get worse (1 and 2 seconds being typical) but still pretty good. Then, something happened around December 2009, when things got really bad, taking typically more than 5 seconds to load the BookMooch home page.

Initially, I thought it was simply that we needed a new server. So, I upgraded the server, and that helped a little bit, but not nearly enough.

Then I thought that this was caused by the search feature not being fast enough to keep up. So, I rewrote parts of it and removed a few extraneous features that took a lot of time, That speed a search for “Bertrand Russell” from 2.6 seconds to only 0.6 seconds. Much faster!

Unfortunately, that didn’t help either: the web site was still slow.

Then I started looking at the usage logs to see what was going on. I had 16 CPUs on the server, and when I shut BookMooch down, 14 of them were doing full text searches. Hmm.. And… 10 of those searches came from one search engine named cuil.com. Apparently, they’re a brand new search engine, and are trying to distinguish themselves by deeply indexing the web. Their indexer was hitting my full text searching feature pretty hard.

So, I banned cuil.com completely from my site. I also banned Google and Yahoo and Baidu from using the full text feature to index BookMooch. Those sites can (and do) still index BookMooch, but they stay away from pages that are full text searches (indexing book details pages, mostly). A normal page on BookMooch takes about 2/100th of a second of the server’s time, whereas a complicated full text search can take seconds.

Bang! That was it!

The load on the server plummeted, which you can see on this chart:

There are two days of pre-ban usage on the left, where the load is very high, around 17, and then after the ban on the right side, where the server load drops to about 1/4 the previous level, to about 4. The server was a lot less stressed out.

Going back to my chart of how many seconds it takes to load the BookMooch home page, we have good news:

You can see that it used to take more than 5 seconds to load the home page, but for two days now it’s been less than 1/5th of a second to do so.

---

Thanks, everyone, for bearing with me as I figured this all out.

Two changes have occurred to the search results, as part of making searching work faster:

a) the search results page no longer displays 3 “related searches” for you automatically. If you want to see related searches, there’s a “RELATED SEARCHES” button toward the top right of the page.

b) the bottom of the search results page used to show you the other languages that the books from your search were in. This is no longer shown. It was noisy and not very useful and turned out to take a lot of time to calculate. I think most people use “advanced search” if they want to narrow their search down to books of a particular language.

Now that I’ve fixed the speed problems, I’m back on BM, working on new features.

One last piece of good news: now that BM is fast again, usage of the web site has climbed 45% in the past two days.

-john

BookMooch now in Swedish

The BookMooch web site is now available in Swedish at: http://se.bookmooch.com

My plan is to have the BookMooch web site available in lots of other languages… once I replace the Amazon.com book database use by BookMooch with a better and more international one. My intent is to add the OCLC, LibraryThing and OpenLibrary databases as sources of book information to BookMooch. I have agreements already in place with OCLC and OpenLibrary to do this.

Once I have a better and more international database in place, not only will we have much better “meta-information” about our books (including English ones) but it’ll be easy to add books from other languages as well, since BookMooch will know about them. The OCLC database in particular, is strong on Scandinavian books, as they tie into their excellent library system database.

---

The reason I did Swedish right away is that professional translator and fellow moocher Susanne Andersson had a free week between jobs and offered to do it right away.

It’s a huge amount of work (the BookMooch web site is big!), and she did it quickly and was a pleasure to work with. I’m including her business card here to thank her (hint, hint, if anyone needs to translate something into Swedish)

Susanne says that she’d be happy to supervise any translation of documents she didn’t get to, She’d also like to correspond with any other Swedes doing corrections of the translation, so that the same terminology is used throughout.

New server is up

The new server is up and running.

It’s 12:30 in the morning here (I’ve been working on it since 9am), so I’m going to bed now.

I still have more tweaking to do to make BookMooch fast again. I’ll get started on that tomorrow morning.

I *think* what’s taking up most of the machine’s time are full text searches. If that’s the case, I’ll need to look at that a bit further and see if there are any quick fixes I can do to make it faster, or if it’s time to move to a new searching engine for BookMooch.

So… for the time being, please have a bit of patience if the server isn’t as fast as it should be. I’m on it!

-john

BM moving to a new server tuesday

BookMooch will (hopefully) be moving a new server on tuesday.

That means BookMooch will probably be unavailable for quite a few hours tomorrow, as I move the database.

I’m preparing as much as possible ahead of time, to minimize unavailability.

I know that BookMooch has been really slow for the past two weeks, and this should fix the slowness.

Wish me luck!

And thanks for your patience!

-john

Power outage

BookMooch was down for about 7 hours due to a power outage.

More specifically, the circuit breaker tripped on our server rack at the hosting company, because we were using too much electricity.

I’ve noticed that BM has been slow for about a week, and so I’ve been preparing a new server for BookMooch. Last night, I put it into a “stress test” to make sure there weren’t any hardware problems. I’m guessing that the electric power draw of the stress test is what sent the circuit breaker over the edge.

No data loss, everything is back up ok, but I do need to find a way to lower the electrical draw, so I can get the new BM server operational, and everything will get zippy fast again!

-john

Librarymooch ?

Here’s an idea I’d like feedback on: a separate web site just for libraries who want to mooch from each other: librarymooch.com

Yes, I know that Inter-Library-Loan exists, but it’s different from mooching, as it’s not a good way for libraries to permanently acquire books that they’d otherwise want to buy, nor is it a good way to get rid of books they no longer have storage for.

A few libraries already use BookMooch, and supporting their survival and growth is one of my personal goals with BookMooch. However, the needs of libraries are a bit different from the general public, such as:

1) they have a LOT more books, and would rather send a minimum of a box of books, and not one at a time
2) they would prefer to send their books to other libraries
3) they would like BookMooch to be integrated with their existing cataloging software

The goal would be to build a vibrant library community mooching from each other, and since their points would be acquired just as on BookMooch, offer libraries the possibilities of expanding their network (both giving and mooching) by joining the larger BookMooch community.

Thoughts? Any libraries interested in such a thing? (if so, email me).

One possibility is making librarymooch.com simply be a sub-site of BookMooch, where a checkbox toggles which site you’re on, so that it’d be easy for libraries to “upgrade” to the greater bookmooch site, or stay within the smaller library-only community. In this same direction, it could be possible for other sub-groups to make their own “mini mooch”, such as (say) a university that wants to initially just have mooching as a person-to-person activity on their campus.

Not to worry, this isn’t what I’m working on at the moment, it’s just an idea that came up in a conversation today, and I wanted to get feedback on it.

-john