I’ve added a feature that will recommend books that BookMooch thinks you’ll like.

You can find the feature under the “Browse” tab:
Reco5

you will then get personalized recommendations, 20 to a page.

Recopg

Recommendations are built using this formula:

1) find all the books you’ve previously mooched, or have on your wishlist or save for later

2) for each of those books, see what other books are recommended to people who like that book

3) total up all the recommendations, and sort them by frequency (ie, the books most often recommended show up first)

4) hide recommended books that are not available for mooching

5) optionally hide books you click the “hide” button on

When you first use the recommendations, it will tend to give you best-sellers as the top recommendations, because that’s what everyone else is also reading. If you like best-sellers, great! If you don’t, no problem, just click “hide” on the books you don’t want to see, and they’ll be hidden from your recommendations forever (in the future, BookMooch will have a feature where you can unhide recommendations you’ve suppressed).

Your recommendations are saved (cached) in the system for 24 hours, so they will not change for 24 hours even if you add/remove books to your wishlist. This is done so that paging next/previous through the recommendations is quick.


I’ve also added a “recommended” button under each person’s bio page. This will show you the books that are being recommended to this person:

Recopg6

I’m trying to decide whether to keep this ability or not.

My thinking was “hey, I like this person’s book tastes, I wonder what BookMooch would recommend they read? Maybe I’d like the recommendations”

Does that make sense, or is that feature overkill? My wife thinks it’s silly, and I do feel like the “recommended” button is awkward and not intuitive. But, perhaps it’s useful? Does anyone have suggestions for how to make the link less awkward?

Choose a language

October 23, 2007

GloberA week ago, I changed the way BookMooch looked for first-time visitor outside an English-speaking country.

For these other languages, BookMooch would auto-detect the primary language in their country, and send them to the appropriate site.

This has confused some people, most specifically ex-pat Americans living in countries where they don’t speak the local language. And for whatever reason, they aren’t noticing the language choices on the bottom right of the home page.

So, as of now, I’ve changed how this works.

From now on, if you come from a country that isn’t definitely English speaking (USA, UK, Ireland, Australia, New Zealand) you’re presented with a version of the home page that gives you language choices. You can stay in the English site by clicking on any of the tabs, or switch languages by clicking the appropriate language graphic. Here is what it looks like:

Langchoices

Please let me know if there are other countries where we-only-speak-english-here that I should have skip this language-choice page, or if this page could be improved or is confusing.

I’ve re-enabled RSS feeds and book recommendations, though with a few stipulations.

* RSS feeds for wishlists, wishlists-related, and inventory, are all re-enabled. However, they are currently being refreshed every 12 hours. I’ll probably be making the refresh time a lot shorter, once I’m comfortable that it won’t impact the BookMooch web site’s performance.

* RSS feeds for search are no longer available, maybe permanently so. The reason? First, I found that the RSS feed for searches didn’t work right anyway: multiple-word searches didn’t work, and advanced search was iffy. Also, the way the RSS feed was returned was one-document-for-the-entire-search, which wasn’t very helpful, because what people really wanted was just the new hits for the search to be returned, so it would be easy to see what new books were available for that search. Since this RSS feed causes a lot of server load, and doesn’t work right anyway, I’ve retired it for now.

* Recommendations have been re-enabled, but they’re only displayed for books that have already had recommendations calculated for them. Recommendations are no longer calculated on-the-fly, as that’s what caused problems before, in that tons of books needed recommendations, lots of pages were hit in parallel, and wham! the server got overloaded trying to keep up. In the background, I have on “thread” making recommendations for all the books on BM, and once that completes, just about every book on BM will have a “recommended” line

For techies: I previously used a 1 megabyte in-memory cache to store RSS feeds. I found a few problems with this:
1) when I restarted the server, there was nothing in memory, so the server would get hammered for a while
2) a 1mb cache was woefully inadequate, with RSS feeds often being 100k large, that only stores 10 RSS feeds in the cache.

I devised a on-the-filesystem cache, so that an RSS feed is generated the first time it is requested, and saved to disk, whereupon successive requests for that RSS feed simply read the stored XML document. At the time the file is save, I create a scheduled task to delete the file in 12 hours, so that it’ll be regenerated.

I found that after running for 5 hours, 1.4 gigs of data has accumulated in the on-disk RSS cache, which shows that an in-memory cache would really not work. Another interesting fact: 2500 different RSS feeds were requested in that 5h period, so that gives us an idea how many people are using the feeds (not that many, but that’s decent).

Below is a chart of server load, before RSS was enabled, and then after I enabled them. You can see that RSS feeds pounded the server for quite a while, about 1 hour actually, but that once they all got cached, server load when way, way down, and is good now. As long as this continues to be the case, I’ll start to shorten the duration RSS files are cached, so that they’re more up-to-date. You’ll note, however, that the server load is permanently greater after the RSS feeds have been enabled, even after the cache has kicked in. The web site seems very responsive none the less, which is really what I care about.

Sload2

Slow1-1
The recent New York Times article on BookMooch has greatly increased the daily hits and size of the database, and so BookMooch is going to run slowly for a few days.

I’ve ordered more memory for the server, which should solve the speed problem, but it’s “special” memory that needs to be manufactured (because I’m using a 64bit server with 8 CPUs) so it’ll probably take until the end of next week to get the memory upgrade up.

In the meantime, I’ve disabled:
a) RSS feeds (that’s about 50% of the hits on BookMooch these days)
b) Recommended books

That seems to have immediately greatly helped the speed. I have some ideas about how to improve the performance of RSS, so I’m going to try to restore them tomorrow.

Here’s an interesting chart:

Sload

you can see 10 minutes of running BookMooch normally, compared to 10 minutes with RSS disabled. The server load plummets, and the web site is really fast.

-john

Recommendations

October 14, 2007

A few days ago, I blogged about how we’re using Amazon recommendations.

However, the Amazon recommendations are not perfect for a few reasons:
1) a lot of books don’t have any recommendations from Amazon
2) Amazon only gives 5 recommendations, and generally to popular books, which BookMooch doesn’t have available

I spent today devising a recommendation system based on BookMooch’s own data, and it’s now up and running.

Next to most books, you’ll now find a “Recommended:” field, and a drop down list of up to 10 books.

Shop1

The last choice in the drop down goes to “show more recommendations”, which lists up to 40 recommended books:

Shop3

The first 10 recommendations are the same as on the previous page’s drop down list box.

On this page, each recommendation has a HIDE button (if you’re logged in). Click that, and that recommended book will disappear forever from all recommendations. In this way, you can remove books you definitely don’t like, or have already read, from the recommendations BookMooch gives you.

One other note: a lot of the recommended books are very popular, and have long wishlists, and many givers & moochers. The screen looked awful with all that noise, like so:

Precon1

now, any list of user links on the book details that is more than 3 people long is instead displayed as a drop down list box, so the book above now looks a lot cleaner:

Postcon

The recommendations are quite complicated to generate (I’ll explain how they’re created a bit lower in this blog entry). If you go to a book that hasn’t yet had its recommendations generated, the page will take an extra second or two to load as this is done. the result is then saved for 7 days, after which it is refreshed.

Why refresh the recommendations so often? Well, because I wanted the recommendations to suggest books that are available at BookMooch, not that you have to buy. That’s the biggest problem with the Amazon recommendations: 5 suggestions isn’t much to go on.

The BookMooch recommended books have to be available from at least one person in order to be recommended. Of course, that person may not want to send to your country, or may give it away in an hour, but at least this means the recommendations are books that are showing up on BookMooch, and I’m finding this to work quite well, with BookMooch suggesting usually good recommendations that are consistently available, or else recently available.

How the recommendations are built:

* first, a list of members who have this book mooched, given or wishlisted or save-for-later or have in their inventory
* then, all the related editions to the current book are also combed through in this way, to build a big list of “related members” for this book
* for each “related member” we then look through all the books they have mooched, given, wishlisted/saved-for-later or have in their inventory and compile a big list.
* we sort the list of related books by how often they appear in the big list, and discard any book that doesn’t occur at least 5 times
* we remove all books that aren’t currently available, or that have exactly the same book title (ie, alternate editions)

Currently, recommendations for all the books in BookMooch are being built in the background so that there’s no delay when you go to the book the first time. This will take about two weeks, but in the meantime, you will be able to see the recommendations, you just might have to wait a few seconds for the book details page to load.

Yet more new features

October 14, 2007

Yesterday, I went through the “to do” database I maintain for BookMooch. I write every suggestion (that isn’t totally brain damaged) in it.

Here’s what I found:
* I have 957 items on my todo list
* I marked 74 high priority items as completed (yay)
* my “highest priority” list has 101 items on it

For the next few weeks, I’ll be focussing on getting that “highest priority” list down to a manageable size, so you’ll be seeing a lot of blog postings from me, with lots of smaller changes. Virtually all of these ideas come from the BookMooch community — as I said about, I write everything down, but often I can’t get to it for a while, but I do eventually get to many of them.

Here are the changes that went up on BookMooch today:


Japanese

BookMoooch in Japanese is now live and launched. A special thanks to Ben Gerdemann and Benito Cachinero-Gorman for correcting a lot of the translation


Language detection

The BookMooch home page redirects you to the Spanish, German, Italian, Portuguese or Japanese version of BookMooch if your TCP/IP address indicates you are from there. I’m using a database from http://www.ip2location.com/ and it seems very accurate.

Langs4

BookMooch is available in several localized versions:

http://it.bookmooch.com – Italian
http://es.bookmooch.com – Spanish
http://de.bookmooch.com – German
http://pt.bookmooch.com – Portuguese
http://jp.bookmooch.com – Japanese

If you want to use the English version, click on the word “English” on the bottom right of the BookMooch home page. Or just re-enter in your browser http://bookmooch.com

Also note that I depend on you, the BookMooch members, to spot and correct bad translations. The 1st pass of all translations is done by Google, so it’s usually pretty bad, and then people can click the “correct this translation” button and fix it themselves.

Note that the this auto-detection-and-redirection only happens the first time you come to BookMooch, as I give your browser a cookie to indicate that you’ve been auto-detected already. I hope this auto-redirection on the home page won’t confuse any long-time BookMooch users, but I suspect new users will appreciate it.

The reason I did this auto-detection is:

* I get a surprising amount of email from people who don’t even know that the BookMooch web site is available in their native language, because they don’t notice the links on the bottom of the home page (which is the only place it’s mentioned)

* most big web sites, such as Google and Yahoo, also do this. When I was in Stockholm this summer, Google always came to me in Swedish. Unfortunately, Google didn’t make it easy to switch to English (sigh), which is why on BookMooch I try to make it simple.

* for many non-English speakers, when they go to BookMooch and it’s in English, they don’t even bother reading the page: they assume it’s not available in their language


Who am I Smooching?

The “give a smooch” page now displays who you’re giving a smooch To:

Picture 1-5


Wishlist notification order change

People with less than 1 point, are now notified last of a book on their wishlist.

Hmmmm.. I just realized that people on “vacation” should probably also be notified last of the availability of a book on their wishlist.


Books to send

On the member main menu, if you have any books you need to send, this has always been displayed in bright red:

Ms1

As of today, if you “delay” all the books you need to send, the message isn’t in bright red. It’s a very small thing, I know, but since I’m sometimes carrying books back with me when I fly from the UK to the USA (and back) I wanted a way to get nagged only if I had un-attended books to send. Now it looks like this if you only have “delayed” books to send:

Ms2


Mooch confirm security test

Tech support received several emails recently from people saying there was no security test on the “who do you want to mooch from?” page. The was there, but the reason they couldn’t see it, I think, is that it was at the bottom of the page, after the choices of people to mooch from. If the list was very long, they didn’t notice it at all, especially since the “mooch” button was at the top of the form.

I’ve now moved the “security test” field to the top of the “who do you want to mooch from?” page, though in many cases it won’t appear at all, as it is unneeded, thanks to a change recently.


Date joined

The date the person became a member of BookMooch is now displayed in their bio page:

Djo


Chart data fixed

My “statistics charts” at http://bookmooch.com/about/stats collect data each night for the previous day. Over the first year of BookMooch, there were various bugs, and sometimes the stat missed a day, and sometimes a stat didn’t work for a while, and missed a few months (whoops!).

This is why some charts had very steep lines like this one:

Tbm

If you look at the X axis, you’ll notice that the labels are evenly spaced in 2 month increments, except for the last one, which shows five months. The reason, is that the “total books mooched” stat wasn’t being collected for 3 months, so when I fixed it, it showed a huge jump.

Yesterday, I wrote a program to fill in the missing data points, with the average between the two points we do have. This fixes the charts and gives a now realistic picture of BookMooch’s growth:

Tbm2

Note that the down blip on this chart is real: it was caused by the database crash we had at that time, and we did “lose” a small number of mooches because of it.

A few small feature updates and bug fixes today:

* allow removing book from wishlist/save-for-later/inventory from details page. When looking at a book details page, you’ll now see a button appear for “remove from inventory” “remove from save-for-later” and “remove from wishlist” when those are relevant.

– the “add to wishlist” and “add to inventory” buttons in the moochbar, and elsewhere, have been re-titled to put the important word in front. It’s bad grammar, but people occasionally click the wrong button so now you will see buttons occasionally like “wishlist add” and “inventory add” — hopefully this will have a small improvement on people clicking the right button.

Bugs:

– a rare problem where you couldn’t cancel a mooch that the other person wasn’t replying on, was fixed

– it’s possible for you to upload a photo of yourself and you don’t see it right away. This is because your web browser has cached it. I’m not seeing this too often, and I’m not sure I can do much about it, but the problem likely came out because I changed the way the user photos work, so that they’re about 200x faster to fetch. This is a prelude to having the ability to upload book covers to BM.

Amazon recommends

October 9, 2007

A few months ago, Amazon added “similar items” to their data feed. This wasn’t available when BookMooch launched, and I’ve been itching to offer recommendations on BookMooch for some time now.

Book detail pages now show a “Amazon recommends” section:

Reco1

which simply displays the book titles:

Reco2

BookMooch has 750,000 books in its database, so it’s going to take a while to get all the recommendation data from Amazon. The way I’ve got it working now, is that when you look at a book details page and there is no recommendation data yet for it, BookMooch ads it to a background task to add it soon, and usually in a few minutes it’ll be there.

It turns out that adding recommended books will greatly increase the size of the BookMooch database, because each time a book is recommended, BookMooch needs to fetch all the book data for that recommended book, and store that into BookMooch. And of course, we then need to do the same thing to the books that were recommended by the recommendation, and so on. Sometimes, such as with children’s books, which are very popular already on BookMooch, all the books are already in the BookMooch database. Other times, such as with text books, I see a spiraling process of adding all the recommended books in a field. It’ll be interesting to see how this plays out.

In a week or two, I’ll start adding “recommendations” from your wishlist, inventory, pending and mooch history, which will find books that Amazon recommends that are also *available* for Mooching right now on BookMooch. I’ll also, in a secondary list, show you recommendations that aren’t currently moochable.

I may also change the “Amazon recommends” list to bold or otherwise mark the recommended books which are available for mooching, so you can easily scan it for books you can get right now.

Flame-1
There has been a lot of debate about the new wishlist notification system and I wanted to post my thoughts.

Yes, I’ve read all the messages.

I’m not going to change the notification system back at this time.

My plan is to wait 6 months, and then re-evaluate and see if we’d be better served with a different strategy, and that wouldn’t necessarily be the old way, it could be something brand new.

The Internet is changing really, really quickly, and BookMooch is growing leaps and bounds.

With so many communication methods (web, email, RSS, IM, SMS) I expect that the whole topic of “fairness for contended resources” will have to be regularly revisited.

For example, one of the other ideas I’ve thought about is Disney’s FastPass

Each person would be able to “jump the queue” on one book.

Here is how it would work:
– you pick one book at a time that you are first in line for
– you can leave that line and go to another
– when you get that book, you “cash in” your fast pass

That way, everyone could pick the one book they want the most, and have the best chance of getting it.

Instead of a email notification, the book would be automatically “reserved” for them for 24h, and no-one else could mooch it for a day.

I don’t know whether
1) you automatically get another fastpass as soon as you use one up
2) or, you get a limited number of fastpasses, either an idea like “one per month” or “one for every 10 books you give away”


Anyhow, this idea is also a long, long way from being implemented on BookMooch, it’s just one of many alternative ideas I have about how this might work.

At some point, when non-email forms of checking the site for available books are quite popular, I’ll need to reconsider and look at other ways to deal with “fairness for contended resources” problem.

Right now, I’m thinking some sort of “on hold” or “reserved” system is where this will go, but I’m not going to give this much more thought for at least 6 months. There are too many other features to add to BookMooch right now.

Wishlist notifications

October 6, 2007

Wsls

The way in which the wishlist email notifications works has now changed.

The way it works as of right now is:

1) when a book is suddenly available, and many people have that book on their wishlist, the people to notify are shuffled randomly (and reshuffled each time another copy is made available)

2) the first person on the notification list gets a message exactly 2 minutes after the book is added

3) each successive notification goes out 4 hours later. The notifications for the book stop being sent out if the book has been mooched and is no longer available, so people aren’t notified of a book that is no longer actually available.

Why the change? I wanted to fix these long-standing complaints:

1) if someone mistakenly adds a book and then removes it, the notifications have already gone out, and everyone who then clicks to mooch the book is annoyed that it isn’t available. With the new system, if the person removes the book (ie, clicks the “whoops” button) within 2 minutes, then no notifications are sent.

2) if someone mooches the book, everyone else gets notified but they’re too late, so they’re upset and annoyed by always being told that they’re too slow. This is an especially frequent occurrence with new and popular books, that can have 50 people wanting the same book.

3) the old system favored people who were always on their email program, especially if they set their mobile phone up to receive their wishlist notifications. That wasn’t really fair to the less-wired BookMooch users.

4) Europeans trying to get American books were at a serious disadvantage, as they are often sleeping when the book was added. The 4h “window” they have to grab the book under the new system may still find them sleeping, but it’s better than nothing. I’ve been thinking about an 8h window between notifications, but that seems awfully long. A 4h space between notifications will give each person a decent chance of receiving the email and getting the book.

5) My goal with the new system is for everyone to have an equal chance of getting a wishlisted book, but the “equal chance” now comes from the random shuffle of notifications, rather than the “first come, first served ‘free for all'” that was the previous system.

Note that a wishlisted book will be seen as available on the BookMooch web site, so if someone is sitting on BookMooch continually reloading their wishlist page, they’ll notice the book being available before anyone else. Hopefully, not that many people will do this, and if you set up a robot program to do this, I may get upset.

RSS feeds of your wishlist will show the book as available “pretty-much” right away, though RSS feeds are a on a five-minute delay, to help combat this and make the new system more fair, so that the first person notified still has a few minutes’ notice before anyone else gets a notification.

On my todo list is to add Instant Messenger support to the notifications, so you don’t have to be on email to get your timely notification.