Posts tagged recsys

The Million Song Dataset Challenge is a contest hosted on Kaggle. Its goal is to predict the songs that 100,000 users will listen to, given their listening history and additional listening histories and data about the songs.

Predicting held-out past user choices is a proxy for another task that cannot be directly evaluated without using a live system: personalized recommendation.

MyMediaLite is a tool/library containing state-of-the-art recommendation algorithms. In this post, I explain how MyMediaLite can be used to make predictions for the Million Song Dataset Challenge.

Preliminaries

First, you need to install MyMediaLite. Don’t worry, it is quite easy, and should work fine on Linux, Mac OS X, and Windows.

You will also need several gigabytes of disk space, the challenge datasets, and a working Unix-like environment. On Linux and Mac this should not be a problem. For Windows you could use Cygwin to get such an environment.

In the following, I assume that you have installed MyMediaLite 3.01 (it must be the latest version, because only that one contains some features we will make use of) in ~/src/MyMediaLite. If it is somewhere else, just adapt the paths below accordingly.

Data Preparation

In the MyMediaLite directory, create a directory data/millionsong, and put the unzipped competition dataset there.

cat kaggle_users.txt | perl -ne 'chomp; print "$_\t" . ++$l . "\n"' > user_id_mappings.txt
cut -f 2 user_id_mappings.txt > test_users.txt
cut -f 2 -d ' ' kaggle_songs.txt > candidate_items.txt

# create dataset
~/src/MyMediaLite/scripts/import_dataset.pl --load-user-mapping=user_id_mappings.txt --load-item-mapping=kaggle_songs.txt kaggle_visible_evaluation_triplets.txt > msd.train.txt

# create CV splits
mkdir cv
~/src/MyMediaLite/scripts/per_user_crossvalidation.pl --k=5 --filename=cv/msd < msd.train.txt

# use one split for validation
cp cv/msd-0.train.txt msd_validation.train.txt
cp cv/msd-0.test.txt msd_validation.test.txt
mkdir validation_predictions
mkdir validation_submissions

# prepare directories for prediction/submission files and logs
mkdir logs
mkdir submissions
mkdir predictions

Trying out Different Recommenders

Run in the MyMediaLite directory:

bin/item_recommendation --training-file=msd_validation.train.txt --test-file=msd_validation.test.txt --data-dir=data/millionsong --recommender=MostPopular --random-seed=1 --predict-items-number=500 --num-test-users=1000 --no-id-mapping --candidate-items=candidate_items.txt

You will get an output like this:

Set random seed to 1.
loading_time 1.67
memory 21
training data: 110000 users, 149052 items, 1160746 events, sparsity 99.99292
test data:     110000 users, 77330 items, 290187 events, sparsity 99.99659
MostPopular 
training_time 00:00:00.0718350 .AUC 0.56605 prec@5 0.0078 prec@10 0.007 MAP 0.02051 recall@5 0.01875 recall@10 0.03011 NDCG 0.05008 MRR 0.02324 num_users 1000 num_items 386213 num_lists 1000 testing_time 00:00:35.3801840
memory 120

The MAP 0.02051 is the interesting piece of information: This is an estimate of how well we will perform on the leaderboard with this recommender.

The command for the WRMF recommender is similar, only that we also see results at different iterations:

k=28; cpos=28; reg=0.002; bin/item_recommendation --training-file=msd_validation.train.txt --test-file=msd_validation.test.txt  --recommender=WRMF --random-seed=1 --predict-items-number=500 --num-test-users=1000 --test-users=test_users.txt --find-iter=1 --max-iter=30 --recommender-options="num_iter=0 num_factors=$k c_pos=$cpos reg=$reg" --data-dir=data/millionsong --no-id-mapping --candidate-items=candidate_items.txt

The output will be like this (I removed some parts for better readability):

WRMF num_factors=28 regularization=0.002 c_pos=28 num_iter=0
MAP 0.00003 iteration 0
MAP 0.01106 iteration 1
MAP 0.01659 iteration 2
MAP 0.02593 iteration 3
MAP 0.03558 iteration 4
...
MAP 0.05341 iteration 30

Nice. This is already some improvement over the MostPopular baseline.

Creating a Submission

bin/item_recommendation --training-file=data/millionsong/msd.train.txt --recommender=MostPopular --predict-items-number=500 --prediction-file=data/millionsong/predictions/mp.pred --test-users=data/millionsong/kaggle_users.txt

k=28; cpos=28; reg=0.002; it=30; bin/item_recommendation --training-file=msd.train.txt --recommender=WRMF --random-seed=1 --predict-items-number=500 --recommender-options="num_iter=$it num_factors=$k c_pos=$cpos reg=$reg" --prediction-file=predictions/wrmf-k-$k-cpos-$cpos-reg-$reg-it-$it.pred --test-users=kaggle_users.txt --candidate-items=candidate_items.txt --data-dir=data/millionsong

MyMediaLite’s output format is a bit different from the submission file format, so I wrote a little script to convert the prediction file:

~/src/MyMediaLite/scripts/msdchallenge/create_submission.sh < predictions/wrmf-k-28-cpos-28-reg-0.002-it-30.pred > submissions/wrmf-k-28-cpos-28-reg-0.002-it-30.sub

~/src/MyMediaLite/scripts/msdchallenge/create_submission.sh < predictions/mp.pred > submissions/mp.sub

It will not hurt to make sure the submission file is in the correct format (using the script provided by the organizers) before trying to upload it:

./validate_submission.py submissions/wrmf-k-28-cpos-28-reg-0.002-it-30.sub

./validate_submission.py submissions/mp.sub

Compress before upload:

gzip submissions/wrmf-k-28-cpos-28-reg-0.002-it-30.sub

gzip submissions/mp.sub

Submission

Now you can upload the submission files to Kaggle. I got the following results:

• MostPopular: 0.02255
• WRMF: 0.05654

Next Steps

I am currently preparing three further blog posts, which I will publish during the next days (links will be provided when the post are ready):

1. using song attributes (artists)
2. blending results from different recommenders
3. using additional interaction information from the Million Song Dataset

The approach demonstrated here is just a simple one, relying on functions that are already available in MyMediaLite. One can think of many extensions, either using existing functionality, or implementing them using the framework provided by MyMediaLite:

1. Improving the BPR-MF results published by the contest organizers in their AdMIRe paper.
2. Take counts into account: in the popuarity model, in WRMF, in BPR-MF

1. Million Song Dataset: Breaking the Collaborative Filtering Ceiling
2. Quora: How does Last.fm compute lists of similar artists?
3. Cold Hard Facts: The Million Song Dataset Challenge: Part I

Want to learn more about MyMediaLite?

Have a look at the website, browse the API documentation and browse the source code on GitHub, search the Google Group archives.

Questions? Problems?

In case there are questions, do not hesitate to ask them in MyMediaLite’s Google Group or in the competition forum.

I handed in my thesis in February, subsequently terminated my work contract with the university, and went for three nice trips, one to Cape Town, one to Southern Germany, and one to Hamburg.

I would like to thank all the people who helped me during with my job hunting, by giving hints and providing introductions and recommendations. I also thank the people and companies that invited me for an interview.

The KDD Cup 2012 has recently started, and there are already more than 200 participating teams. This year, the competition is hosted by Kaggle.

Because there were several questions in the forum about the evaluation measure MAP@3, I hereby publish my validation script, hoping that it may be useful.

The script expects two files, one prediction file and one file containing the ground truth (actual clicks). The prediction file is similar to the submission file format - the only exception is that it should contain exactly one recommendation list per user. The ground truth file is a “standard” rating file, just like rec_log_test.txt and rec_log_train.txt.

How to call the script:

./evaluate.pl —prediction-file=pred —groundtruth-file=rec_log_train.last0.1.txt

I created the file rec_log_train.last0.1.txt using the command

tail -n 7320927 rec_log_train.txt > reg_log_train.last0.1.txt

The script is written Perl, which should be installed by default on a typical Linux or Mac OS X machine. You can download it from GitHub: https://github.com/zenogantner/MyMediaLite/blob/master/scripts/kddcup2012/evaluate.pl

Of course, I cannot guarantee that it works correctly. If you have questions or suggestions for improvement, do not hesitate to contact me.