Finally!

April 18th, 2008

It came to me like a “oh my god” moment today! I know what my thesis will be about. The semantic web. Specifically, Semantic Web Services. This is why i chose the topic to pursue, yea i know the topic is still too broad but at least I have an understanding what im finally doing and why.

This is how it came about. I was given a task at work, its a prototype application using Flex and AIR and I started to look into how these particular technologies work with XML and web services. I started to look into RDF parsing and found that there is no clear RDF or OWL parser for Action Script 3, that got me thinking and soon had me looking over my class notes on the subject. That then brought me to where im at now, how these technologies work with web services.

The technology to work with web services isnt that difficult but the technology to have web services find each other to facilitate each others needs captures my attention. The underlining idea of self discovery on the computational level is interesting. How can you have a program locate and basically build itself to accomplish a task? In other words, how can one person locate two people across town, that dont know each but posses skills to complete a task, come together? Before the advent of the web, we used the phone book now we do a quick google search. This is the idea behind the semantic web service, from what i’ve gathered so far. (I might retract after i read a bit more on the subject)

Im going to focus on, first the technology RDF, OWL, see whats currently available and working in production, then im going to focus my attention on roadmaps on how to set up a very simple semantic web service (when i mean simple i mean simple). Finally im going to look at soa type of set ups for sws. (this roadmap might change dramatically as i start to dig deeper. Ill keep the Research section updated with my findind though and ill wrap up all the research related tutorials ive started aswell soon.)

Armando Padilla

Semantic Web – Possible Thesis.

April 12th, 2008

A quarter or two ago I took a graduate class where we had to chose a topic that we possibly would enjoy looking into. Ashok and myself spent three months putting together a 58 page report on the Semantic Web. Its current state, current technology, its challenges, and its future.

If you do a quick google search on the semantic web RDF, OWL, and some other XML based junk will come up. All that technology is just to put meaning into all the documents published on the net, information that gadgets create so one day all that data can be combined to help machines help us. So the theory goes.

Here is the paper.
Semantic Web Paper – by Armando Padilla & Ashok Sahu

Comparing Distance Formulas. Part III

February 13th, 2008

This article will cover the Euclidean Formula. I will cover the basic equation for the algorithm, provide running and working examples for both Ruby and PHP5, but will state advantages and disadvantageous of the algorithm after
covering every algorithm in the article.

Background – Euclidean Algorithm
One of the oldest and one of the most widely used distance formulas is the Euclidean formula. The formula captures two items and compares each attribute of each item to one another to determine how close, related, they
are. In other words. With the formula we can take two users run through each of their characteristics, in our case movie ratings, and determine how similar they are.

But how is the value that the algorithm provides us used? Once the calculations are done between two users we are presented with a value between 0 and infinity. The larger the value the farther apart both users are to one another (not similar). A few example of what values may be provided are 0.56, 2.89, 4. These are the values that will be used to determine how similar both users are to each other.

The Math Equation

The equation above gives us both the short hand and the expanded version. We’ll take a look at the short hand. In short hand the equation takes the difference between the attributes of two objects specified in the (Pi-Q1) portion and squares it. For a single dimension, attribute, the equation will take the square-root and then finish. Fortunately for us, the equation allows us to use more than one attribute so it continues by taking the sum of the next set of attribute differences until there are no additional attributes to compare.

Calculating Distances with an example.
Using the notation above, lets take a simple example using our data set to make it clear what the equation is doing. For this example well use 3 users and 2 movies all users rated.

1:
30878,4,2005-12-26
1481961,2,2005-05-24
885013,4,2005-10-19

5:
30878,3,2004-10-19
1481961,2,2005-09-27
885013,5,2005-05-15

The data sets we are using for this example has three users, user with id 30878, id 1481961, and id 885013 rating two movies. One movie with id 1 and the other with id 5. User 30878 gave the movie 1 a rating of 4 and gave movie 5 a rating of 3. User 1481961 gave movie 1 a rating of 2 and gave a rating of 2 to movie 5. User 885013 gave movie id 1 a rating of 4 and a rating of 5 to movie id 5.

Preparing the above data to use with the Euclidean formula, we convert each user to either a P or Q. The goal is to determine to distance between user 30878 and both 1481961 and 885013. 30878 will be represented by the letter P and Q will be the user 1481961 for the first distance calculation. The second distance calculations will simply change the Q to represent the user 885013. Also, since there are only two movies, dimensions, we know that the equation will take the calculation for 2 dimensions.

Distance between user 30878 and 1481961

Euclidean User Distance 1

Distance between user 30878 and 885013

Euclidean User Distance 2

After calculating the distance between both users what does the final numerical value represent? It simply tells us that the distance, or the similarity between both users is 2.23. The rule of thumb is, as the results decrease in value the similarity between each user increases. If the result was a 3.5 or 2.89 the similarity between the users would decrease while a result of 0 or 0.2 would indicate an extremely similar user.

Visualizing Similarity with a Graph
Yet another way to view the data is graphically. Plotting the users on a X/Y graph where the Y-axis represents one dimension, ratings for video 5, and the X-axis represents ratings for video 1 we can visually see how close the users are to one another.

Euclidean Distance Graph

User 30878 has the coordinates, (4,3), while user 1481961 has coordinates (2,2), and user has coordinates (4,5). The graph describes how close, similar, each user is to the user 30878. User 1481961 is closer to 30878 compared to 885013, thereby making these two users similar.

The Code

PHP5 Class

/**
* Distance Formula Class
*
* @author Armando Padilla, mando81@prodigy.net
* @copyright Armando Padilla.
*
*/
class DistanceFormulas {

public function __construct(){}

public function euclidean(array $objectXAttribute, array $objectYAttribute){

$distanceFromXToY = 0;
for($i=0; $i<count($objectXAttribute); $i++){

$p = $objectXAttribute[$i];
$q = $objectYAttribute[$i];

$distanceFromXToY += pow(($p-$q), 2);

}

return $distanceFromXToY;

}//End

}//End class

PHP5 Test

/**
* Simple example using the DistanceFormula Class
*
* @author Armando Padilla, mando81@prodigy.net
* @copyright Armando Padilla
*/
require_once(“DistanceFormulas.php”);

//List of users along with their ratings.
//usually you would have this in a database and have more than 3 users :-)
//
// rating order: “The Abyss”, “Hunt for Red October”, “Goonies”
$profiles = array(array(“username” => ‘Armando’, “movieratings” => array(5, 4, 4)),
array(“username” => ‘Snoopy’, “movieratings” => array(2,2,5)),
array(“username” => ‘Pearl’, “movieratings” => array(1,4,1)));

//Instantiate the class
$DistanceFormula = new DistanceFormulas();

//Let’s get the distance from Armando to the other two users.
for($i=1; $i<count($profiles); $i++){

$distance .= “Distance from “.$profiles[0]['username']. ” to “.$profiles[$i]['username'].”:: “;
$distance .= $DistanceFormulas->euclidean($profiles[0]['movieratings'], $profiles[$i]['movieratings']);
$distance .= “<br>”;

}

Ruby Example – coming soon.

Comparing Distance Formulas. Part II

February 11th, 2008

Before diving into the algorithms I would like to step back and take a look at the
data we will be analyzing with our examples.

Where did the data come from?
The data originated from the Netflix.com data mining challenge. The challenge was created by Netflix 2-3 years ago to allowed data mining junkies the ability to help the movie renting population find movies they might enjoy based on their current renting selection and similar users with similar renting habits. Of course every junkie needs an incentive so Netflix.com is offered $1,000,000 to anyone that could increase the recommendation engines success rate by 10%. The challenge
is ongoing and ends 2010. More information can be found here

The Data
Focusing on the data, the files contain a list of movies, roughly 17,000, along with 547 user ratings per movie. The format that Netflix has placed the data
into is:

1:
1488844,3,2005-09-06
822109,5,2005-05-13
885013,4,2005-10-19
30878,4,2005-12-26
823519,3,2004-05-03
893988,3,2005-11-17
124105,4,2004-08-05
1248029,3,2004-04-22

Where 1 is the movie id. The movie id is a unique number which represents a movie title in their system. For example, the movie id shown above, 1, can be associated to the movie, The Abyss. Since the movie id is unique to one and only one movie we can safely assume that The Abyss will always be 1.

The following lines contain user rating information. Taking one line as an example, we can break up the data by commos. The first piece of the line is the unique id of a user, in this case a person using Netflix to rent a movie. Again, we can safely
assume that one id is associated to one and only one user in the system. As an
example. The id, 30878, is associated to the user, Armando Padilla.

The next item on the line is the rating that the user has given to the movie. In
the data that Netflix.com has used the ratings can be anywhere from 1 to 5, where 1 s a very low rating, indicating that the user hated the movie while 5 is a very high rating indicating that the user has extremely enjoyed watching the movie. Again with an example, the user 30878 has rated The Abyss with a 4, indicating that he extremely enjoyed it.

Finally the last item on the line is the date the user decided to rate the movie.
The format the date is in is, Year-month-dayofmonth. The final reading of a line
will be, “The user, Armando Padilla, with the id of 30878, rated the movie The Abyss with the id of 1 a 4 on December 26th 2006.

How are we using the data for our examples?
As mentioned before there are a little over 17,000 movies in the training data. The specific number of movies rated is 17,770 according the to “readme” file the tranining data is accompined with and the total number of users with movie ratings is 480,189.

For our example we will use all 17,770 movies as attributes for all 480,189 users in the system.