No, I dont mean i dont know how to use it. Ive created a few example projects already but just dont get why the the developers decided to change the syntax SO MUCH. Its like …what??? why??? just for the hell of it??? Also one of the selling points for scala is how well it plays with Java. Most of you know from previous articles how i think Java is not for the web so I dont get why scala would be any different. It runs on the JVM and list runs on a servlet container (ex Tomcat). So all the headaches that i had with Tomcat, Java, and the web application are back…great. So far these are my first impressions of both scala and lift. I plan to create a small TO-DO list using both technologies and publish a small tutorial soon.

It was either scala/lift or learn Ruby well. I chose something completely out in left field. We’ll see, heck Scala+Lift is bosting thats it a few times faster than Ruby on Rails. So…

For those interested in the netflix api check this page out.  It’s basically the netflix documentation. Its pretty straight-forward and should be an easy and fun read.  I was surprised they opened up their ‘predicted rating’ api. Nice touch I can see A LOT of things developers can do with just the prediction REST api.

And for those that want to check out the Zend_Service_Netflix code i currently have, here is the Google Code Project Page link.  

I plan to finish the package by Sunday evening.  Here is my attack plan.

Attack Plan:

1. Finish the concrete classes. (ETA Friday evening) 
2. Finish Up the remaining REST calls (ETA Saturday Evening)
3. Create test using PHPUnit. (ETA. as I go)
4. Documentation.  (ETA Sunday evening) 

Since i see no point in reinventing the wheel, I found Gnosis and SemanticHacker.com.  Gnosis has a great natural language processor and can accomplish the requirement to find “interesting” items but dosnt provide useful articles.   Gnosis instead send you to a google result page (pretty lame but its a start).  On the other hand SemanticHacker.com has come up with a API that basically does both the Natural Language Processing and the relevance articles but has two drawbacks.  1.  The NLP sorta sucks, well its not as good as the Gnosis one.  2.  Articles of interest are mined from Wikipedia which might not be up-to-date or might be misleading.  3.  No FIrefox plugin.

So what am i going to do now? Easy.  I will check out the API that SemanticHacker has released and work on two things.  1.  Create a Firefox plugin for it.  and 2. Expand on the Knowledge Base the system references.  Move from a wikipedia setting to a flickr (look for tagged relevant images) and a news RSS feeds.

Armando Padilla

So I started to draw up some plans during lunch to create a plugin for Firefox that would run through a document opened on the browser.  It will then use a natural language processor to find key words; places, events, etc etc.  I then decided to use the API provided by these folks to find related articles on the subject.  Im not sure if this has been done but I have yet to see it done.  yes I know there are keyword ad related scripts on some sites but i havent seen anything that would compliment articles im reading, movies im watching, or basically anything out there.

Time to get back to work.
Armando Padilla

This is how it came about. I was given a task at work, its a prototype application using Flex and AIR and I started to look into how these particular technologies work with XML and web services. I started to look into RDF parsing and found that there is no clear RDF or OWL parser for Action Script 3, that got me thinking and soon had me looking over my class notes on the subject. That then brought me to where im at now, how these technologies work with web services.

The technology to work with web services isnt that difficult but the technology to have web services find each other to facilitate each others needs captures my attention. The underlining idea of self discovery on the computational level is interesting. How can you have a program locate and basically build itself to accomplish a task? In other words, how can one person locate two people across town, that dont know each but posses skills to complete a task, come together? Before the advent of the web, we used the phone book now we do a quick google search. This is the idea behind the semantic web service, from what i’ve gathered so far. (I might retract after i read a bit more on the subject)

Im going to focus on, first the technology RDF, OWL, see whats currently available and working in production, then im going to focus my attention on roadmaps on how to set up a very simple semantic web service (when i mean simple i mean simple). Finally im going to look at soa type of set ups for sws. (this roadmap might change dramatically as i start to dig deeper. Ill keep the Research section updated with my findind though and ill wrap up all the research related tutorials ive started aswell soon.)

Armando Padilla

If you do a quick google search on the semantic web RDF, OWL, and some other XML based junk will come up. All that technology is just to put meaning into all the documents published on the net, information that gadgets create so one day all that data can be combined to help machines help us. So the theory goes.

Here is the paper.
Semantic Web Paper – by Armando Padilla & Ashok Sahu

Background – Euclidean Algorithm
One of the oldest and one of the most widely used distance formulas is the Euclidean formula. The formula captures two items and compares each attribute of each item to one another to determine how close, related, they
are. In other words. With the formula we can take two users run through each of their characteristics, in our case movie ratings, and determine how similar they are.

But how is the value that the algorithm provides us used? Once the calculations are done between two users we are presented with a value between 0 and infinity. The larger the value the farther apart both users are to one another (not similar). A few example of what values may be provided are 0.56, 2.89, 4. These are the values that will be used to determine how similar both users are to each other.

The Math Equation

The equation above gives us both the short hand and the expanded version. We’ll take a look at the short hand. In short hand the equation takes the difference between the attributes of two objects specified in the (Pi-Q1) portion and squares it. For a single dimension, attribute, the equation will take the square-root and then finish. Fortunately for us, the equation allows us to use more than one attribute so it continues by taking the sum of the next set of attribute differences until there are no additional attributes to compare.

Calculating Distances with an example.
Using the notation above, lets take a simple example using our data set to make it clear what the equation is doing. For this example well use 3 users and 2 movies all users rated.

1:
30878,4,2005-12-26
1481961,2,2005-05-24
885013,4,2005-10-19

5:
30878,3,2004-10-19
1481961,2,2005-09-27
885013,5,2005-05-15

The data sets we are using for this example has three users, user with id 30878, id 1481961, and id 885013 rating two movies. One movie with id 1 and the other with id 5. User 30878 gave the movie 1 a rating of 4 and gave movie 5 a rating of 3. User 1481961 gave movie 1 a rating of 2 and gave a rating of 2 to movie 5. User 885013 gave movie id 1 a rating of 4 and a rating of 5 to movie id 5.

Preparing the above data to use with the Euclidean formula, we convert each user to either a P or Q. The goal is to determine to distance between user 30878 and both 1481961 and 885013. 30878 will be represented by the letter P and Q will be the user 1481961 for the first distance calculation. The second distance calculations will simply change the Q to represent the user 885013. Also, since there are only two movies, dimensions, we know that the equation will take the calculation for 2 dimensions.

Distance between user 30878 and 1481961

Distance between user 30878 and 885013

After calculating the distance between both users what does the final numerical value represent? It simply tells us that the distance, or the similarity between both users is 2.23. The rule of thumb is, as the results decrease in value the similarity between each user increases. If the result was a 3.5 or 2.89 the similarity between the users would decrease while a result of 0 or 0.2 would indicate an extremely similar user.

Visualizing Similarity with a Graph
Yet another way to view the data is graphically. Plotting the users on a X/Y graph where the Y-axis represents one dimension, ratings for video 5, and the X-axis represents ratings for video 1 we can visually see how close the users are to one another.

User 30878 has the coordinates, (4,3), while user 1481961 has coordinates (2,2), and user has coordinates (4,5). The graph describes how close, similar, each user is to the user 30878. User 1481961 is closer to 30878 compared to 885013, thereby making these two users similar.

The Code

PHP5 Class

/**
* Distance Formula Class
*
* @author Armando Padilla, mando81@prodigy.net
* @copyright Armando Padilla.
*
*/
class DistanceFormulas {

public function __construct(){}

public function euclidean(array $objectXAttribute, array $objectYAttribute){

$distanceFromXToY = 0;
for($i=0; $i<count($objectXAttribute); $i++){

$p = $objectXAttribute[$i];
$q = $objectYAttribute[$i];

$distanceFromXToY += pow(($p-$q), 2);

}

return $distanceFromXToY;

}//End

}//End class

PHP5 Test

/**
* Simple example using the DistanceFormula Class
*
* @author Armando Padilla, mando81@prodigy.net
* @copyright Armando Padilla
*/
require_once(”DistanceFormulas.php”);

//List of users along with their ratings.
//usually you would have this in a database and have more than 3 users
//
// rating order: “The Abyss”, “Hunt for Red October”, “Goonies”
$profiles = array(array(”username” => ‘Armando’, “movieratings” => array(5, 4, 4)),
array(”username” => ‘Snoopy’, “movieratings” => array(2,2,5)),
array(”username” => ‘Pearl’, “movieratings” => array(1,4,1)));

//Instantiate the class
$DistanceFormula = new DistanceFormulas();

//Let’s get the distance from Armando to the other two users.
for($i=1; $i<count($profiles); $i++){

$distance .= “Distance from “.$profiles[0]['username']. ” to “.$profiles[$i]['username'].”:: “;
$distance .= $DistanceFormulas->euclidean($profiles[0]['movieratings'], $profiles[$i]['movieratings']);
$distance .= “<br>”;

}

Ruby Example – coming soon.

Where did the data come from?
The data originated from the Netflix.com data mining challenge. The challenge was created by Netflix 2-3 years ago to allowed data mining junkies the ability to help the movie renting population find movies they might enjoy based on their current renting selection and similar users with similar renting habits. Of course every junkie needs an incentive so Netflix.com is offered $1,000,000 to anyone that could increase the recommendation engines success rate by 10%. The challenge
is ongoing and ends 2010. More information can be found here

The Data
Focusing on the data, the files contain a list of movies, roughly 17,000, along with 547 user ratings per movie. The format that Netflix has placed the data
into is:

1:
1488844,3,2005-09-06
822109,5,2005-05-13
885013,4,2005-10-19
30878,4,2005-12-26
823519,3,2004-05-03
893988,3,2005-11-17
124105,4,2004-08-05
1248029,3,2004-04-22

Where 1 is the movie id. The movie id is a unique number which represents a movie title in their system. For example, the movie id shown above, 1, can be associated to the movie, The Abyss. Since the movie id is unique to one and only one movie we can safely assume that The Abyss will always be 1.

The following lines contain user rating information. Taking one line as an example, we can break up the data by commos. The first piece of the line is the unique id of a user, in this case a person using Netflix to rent a movie. Again, we can safely
assume that one id is associated to one and only one user in the system. As an
example. The id, 30878, is associated to the user, Armando Padilla.

The next item on the line is the rating that the user has given to the movie. In
the data that Netflix.com has used the ratings can be anywhere from 1 to 5, where 1 s a very low rating, indicating that the user hated the movie while 5 is a very high rating indicating that the user has extremely enjoyed watching the movie. Again with an example, the user 30878 has rated The Abyss with a 4, indicating that he extremely enjoyed it.

Finally the last item on the line is the date the user decided to rate the movie.
The format the date is in is, Year-month-dayofmonth. The final reading of a line
will be, “The user, Armando Padilla, with the id of 30878, rated the movie The Abyss with the id of 1 a 4 on December 26th 2006.

How are we using the data for our examples?
As mentioned before there are a little over 17,000 movies in the training data. The specific number of movies rated is 17,770 according the to “readme” file the tranining data is accompined with and the total number of users with movie ratings is 480,189.

For our example we will use all 17,770 movies as attributes for all 480,189 users in the system.

What I’ll cover in this first part will be a brief intro into the overall plan. What algorithms will be covered and what data set I will use.

Overall plan
I was given the task of creating a simple but effective, match.com-like, compatibility application. The user would fill out a group of questions and based on the answers the application would present the users overall compatibility to other users in the system. The application was a success and to my surprise resisted the load that 3,000 concurrent users exerted on the application. After launch, I had a lingering question in the back of my mind. How good were those results? Could I have made a better engine?

The goal will be to compare 8 distance formulas and determine which formula produces the best results, which formula produces results efficiently for a web application with 500,000 users (small compared to other data sets out there).

Algorithm’s to Cover
Distance formulas will be the key focus in this piece. I will cover the more traditional distance formulas along with a few obscure ones. The list of algorithms is:

1. Eucledian
2. Manhattan
3. Mahalantos
4. Jacaard
5. Hammin
6. Pearson
7. Sorensen

Of course I will touch on the benefits of each algorithm and the disadvantages that each algorithm contains. I will also have small working Ruby and PHP code to accompany this article.

The hypothetical situation I presented was:

1. Team of 1-10 developers all working on different projects at any given time but return to one main application when the need is there.
2. Application development time is 1/4 th of normal development times.
3. Development team & the companies development teams believe in documentation but never follow through.
4. No QA team. (developer is on the line to deliver on time and with a near flawless product)
5. No unit testing of any kind.

I have, to date, presented this problem to the public but none have come to a conclusion that I see will work. Would a TDD method work? Would Waterfall model wok? Or, would a Spiral model work? Or would Agyle work in this case?

This will be my challenge. Find a software engineering model that works well with such a situation or propose one.