Research Category

Comparing Distance Formulas. Part III

This article will cover the Euclidean Formula. I will cover the basic equation for the algorithm, provide running and working examples for both Ruby and PHP5, but will state advantages and disadvantageous of the algorithm after
covering every algorithm in the article.

Background – Euclidean Algorithm
One of the oldest and one of the most widely used distance formulas is the Euclidean formula. The formula captures two items and compares each attribute of each item to one another to determine how close, related, they
are. In other words. With the formula we can take two users run through each of their characteristics, in our case movie ratings, and determine how similar they are.

But how is the value that the algorithm provides us used? Once the calculations are done between two users we are presented with a value between 0 and infinity. The larger the value the farther apart both users are to one another (not similar). A few example of what values may be provided are 0.56, 2.89, 4. These are the values that will be used to determine how similar both users are to each other.

The Math Equation

The equation above gives us both the short hand and the expanded version. We’ll take a look at the short hand. In short hand the equation takes the difference between the attributes of two objects specified in the (Pi-Q1) portion and squares it. For a single dimension, attribute, the equation will take the square-root and then finish. Fortunately for us, the equation allows us to use more than one attribute so it continues by taking the sum of the next set of attribute differences until there are no additional attributes to compare.

Calculating Distances with an example.
Using the notation above, lets take a simple example using our data set to make it clear what the equation is doing. For this example well use 3 users and 2 movies all users rated.

1:
30878,4,2005-12-26
1481961,2,2005-05-24
885013,4,2005-10-19

5:
30878,3,2004-10-19
1481961,2,2005-09-27
885013,5,2005-05-15

The data sets we are using for this example has three users, user with id 30878, id 1481961, and id 885013 rating two movies. One movie with id 1 and the other with id 5. User 30878 gave the movie 1 a rating of 4 and gave movie 5 a rating of 3. User 1481961 gave movie 1 a rating of 2 and gave a rating of 2 to movie 5. User 885013 gave movie id 1 a rating of 4 and a rating of 5 to movie id 5.

Preparing the above data to use with the Euclidean formula, we convert each user to either a P or Q. The goal is to determine to distance between user 30878 and both 1481961 and 885013. 30878 will be represented by the letter P and Q will be the user 1481961 for the first distance calculation. The second distance calculations will simply change the Q to represent the user 885013. Also, since there are only two movies, dimensions, we know that the equation will take the calculation for 2 dimensions.

Distance between user 30878 and 1481961

Euclidean User Distance 1

Distance between user 30878 and 885013

Euclidean User Distance 2

After calculating the distance between both users what does the final numerical value represent? It simply tells us that the distance, or the similarity between both users is 2.23. The rule of thumb is, as the results decrease in value the similarity between each user increases. If the result was a 3.5 or 2.89 the similarity between the users would decrease while a result of 0 or 0.2 would indicate an extremely similar user.

Visualizing Similarity with a Graph
Yet another way to view the data is graphically. Plotting the users on a X/Y graph where the Y-axis represents one dimension, ratings for video 5, and the X-axis represents ratings for video 1 we can visually see how close the users are to one another.

Euclidean Distance Graph

User 30878 has the coordinates, (4,3), while user 1481961 has coordinates (2,2), and user has coordinates (4,5). The graph describes how close, similar, each user is to the user 30878. User 1481961 is closer to 30878 compared to 885013, thereby making these two users similar.

The Code

PHP5 Class

/**
* Distance Formula Class
*
* @author Armando Padilla, mando81@prodigy.net
* @copyright Armando Padilla.
*
*/
class DistanceFormulas {

public function __construct(){}

public function euclidean(array $objectXAttribute, array $objectYAttribute){

$distanceFromXToY = 0;
for($i=0; $i<count($objectXAttribute); $i++){

$p = $objectXAttribute[$i];
$q = $objectYAttribute[$i];

$distanceFromXToY += pow(($p-$q), 2);

}

return $distanceFromXToY;

}//End

}//End class

PHP5 Test

/**
* Simple example using the DistanceFormula Class
*
* @author Armando Padilla, mando81@prodigy.net
* @copyright Armando Padilla
*/
require_once(“DistanceFormulas.php”);

//List of users along with their ratings.
//usually you would have this in a database and have more than 3 users :-)
//
// rating order: “The Abyss”, “Hunt for Red October”, “Goonies”
$profiles = array(array(“username” => ‘Armando’, “movieratings” => array(5, 4, 4)),
array(“username” => ‘Snoopy’, “movieratings” => array(2,2,5)),
array(“username” => ‘Pearl’, “movieratings” => array(1,4,1)));

//Instantiate the class
$DistanceFormula = new DistanceFormulas();

//Let’s get the distance from Armando to the other two users.
for($i=1; $i<count($profiles); $i++){

$distance .= “Distance from “.$profiles[0]['username']. ” to “.$profiles[$i]['username'].”:: “;
$distance .= $DistanceFormulas->euclidean($profiles[0]['movieratings'], $profiles[$i]['movieratings']);
$distance .= “<br>”;

}

Ruby Example – coming soon.

Comparing Distance Formulas. Part II

Before diving into the algorithms I would like to step back and take a look at the
data we will be analyzing with our examples.

Where did the data come from?
The data originated from the Netflix.com data mining challenge. The challenge was created by Netflix 2-3 years ago to allowed data mining junkies the ability to help the movie renting population find movies they might enjoy based on their current renting selection and similar users with similar renting habits. Of course every junkie needs an incentive so Netflix.com is offered $1,000,000 to anyone that could increase the recommendation engines success rate by 10%. The challenge
is ongoing and ends 2010. More information can be found here

The Data
Focusing on the data, the files contain a list of movies, roughly 17,000, along with 547 user ratings per movie. The format that Netflix has placed the data
into is:

1:
1488844,3,2005-09-06
822109,5,2005-05-13
885013,4,2005-10-19
30878,4,2005-12-26
823519,3,2004-05-03
893988,3,2005-11-17
124105,4,2004-08-05
1248029,3,2004-04-22

Where 1 is the movie id. The movie id is a unique number which represents a movie title in their system. For example, the movie id shown above, 1, can be associated to the movie, The Abyss. Since the movie id is unique to one and only one movie we can safely assume that The Abyss will always be 1.

The following lines contain user rating information. Taking one line as an example, we can break up the data by commos. The first piece of the line is the unique id of a user, in this case a person using Netflix to rent a movie. Again, we can safely
assume that one id is associated to one and only one user in the system. As an
example. The id, 30878, is associated to the user, Armando Padilla.

The next item on the line is the rating that the user has given to the movie. In
the data that Netflix.com has used the ratings can be anywhere from 1 to 5, where 1 s a very low rating, indicating that the user hated the movie while 5 is a very high rating indicating that the user has extremely enjoyed watching the movie. Again with an example, the user 30878 has rated The Abyss with a 4, indicating that he extremely enjoyed it.

Finally the last item on the line is the date the user decided to rate the movie.
The format the date is in is, Year-month-dayofmonth. The final reading of a line
will be, “The user, Armando Padilla, with the id of 30878, rated the movie The Abyss with the id of 1 a 4 on December 26th 2006.

How are we using the data for our examples?
As mentioned before there are a little over 17,000 movies in the training data. The specific number of movies rated is 17,770 according the to “readme” file the tranining data is accompined with and the total number of users with movie ratings is 480,189.

For our example we will use all 17,770 movies as attributes for all 480,189 users in the system.

Comparing Distance Formulas. Part I

Match.com, Singles.com and Netflix.com and countless other sites out there rely on comparing a group of people to answers users submit on a daily basis. Some like Match.com use a long questioner while others, such as Netflix.com, uses renting habits of a user to determine if a the user might like another product. In a nuttshell, How similar two people are depends on how they behave during their visit to the application.

What I’ll cover in this first part will be a brief intro into the overall plan. What algorithms will be covered and what data set I will use.

Overall plan
I was given the task of creating a simple but effective, match.com-like, compatibility application. The user would fill out a group of questions and based on the answers the application would present the users overall compatibility to other users in the system. The application was a success and to my surprise resisted the load that 3,000 concurrent users exerted on the application. After launch, I had a lingering question in the back of my mind. How good were those results? Could I have made a better engine?

The goal will be to compare 8 distance formulas and determine which formula produces the best results, which formula produces results efficiently for a web application with 500,000 users (small compared to other data sets out there).

Algorithm’s to Cover
Distance formulas will be the key focus in this piece. I will cover the more traditional distance formulas along with a few obscure ones. The list of algorithms is:

  1. Eucledian
  2. Manhattan
  3. Mahalantos
  4. Jacaard
  5. Hammin
  6. Pearson
  7. Sorensen

Of course I will touch on the benefits of each algorithm and the disadvantages that each algorithm contains. I will also have small working Ruby and PHP code to accompany this article.