In my previous post, I used this Python script (compliments to Dennis Muhlstein) which uses the Boolean method. The tf-idf method: if a word appears in a text, its term frequency-inverse document frequency is calculated, and that frequency score appears in the vector if a word does not appear in a text, it is represented as a 0. The Boolean method: if a word appears in a text, it is represented simply as a 1 in the vector if a word does not appear in a text, it is represented as a 0. When it comes to vector space modeling, I have seen two common methods: The problem is that there are different ways to represent words as numbers, and it’s never clear which is the best way. But what exactly goes into the vectors in these matrices? Not words from the two texts under comparison, obviously, but numeric representations of the words. In this post, I want to see whether and to what extent different metrics entered into the vectors-either a Boolean entry or a tf-idf score-change the results.įirst, here’s a brief recap of cosine similarity: One way to quantify the similarity between texts is to turn them into term-document matrices, with each row representing one of the texts and each column representing every word that appears in both of the texts. (The matrices will be “sparse” because each text contains only some of the words across both texts.) With these matrices in hand, it is a straightforward mathematical operation to treat them as vectors in Euclidean space and calculate their cosine similarity with the Euclidean dot product formula, which returns a metric between 0 and 1, where 0 = no words shared and 1 = exact copies of the same text. In this case, User #2 won’t be suggested to watch a horror movie as there is no similarity between the romantic genre and the horror genre.In a previous post, I used cosine similarity (a “vector space model”) to compare spoken vs.

Similarly, Suppose User #1 loves to watch movies based on horror, and User #2 loves the romance genre. So the recommendation system will use this data to recommend User #1 to see The Proposal, and Notting Hill as User #1 and User #2 both prefer the romantic genre and its likely that User #1 will like to watch another romantic genre movie and not a horror one.

Taking the example of a movie recommendation system, Suppose one user (User #1) has watched movies like The Fault in our Stars, and The Notebook, which are of romantic genres, and another user (User #2) has watched movies like The Proposal, and Notting Hill, which are also of romantic genres. Where is it used?Ĭosine metric is mainly used in Collaborative Filtering based recommendation systems to offer future recommendations to users. Therefore the points are 50% similar to each other. In the above figure, imagine the value of θ to be 60 degrees, then by cosine similarity formula, Cos 60 =0.5 and Cosine distance is 1- 0.5 = 0.5. Then we can interpret that the two points are 100% similar to each other. Now if the angle between the two points is 0 degrees in the above figure, then the cosine similarity, Cos 0 = 1 and Cosine distance is 1- Cos 0 = 0. Therefore, the shown two points are not similar, and their cosine distance is 1 - Cos 90 = 1. In the above image, there are two data points shown in blue, the angle between these points is 90 degrees, and Cos 90 = 0. Cosine similarity is given by Cos θ, and cosine distance is 1- Cos θ. Thus, Points closer to each other are more similar than points that are far away from each other. As the cosine distance between the data points increases, the cosine similarity, or the amount of similarity decreases, and vice versa. Cosine distance & Cosine Similarity metric is mainly used to find similarities between two data points.

Cosine similarity formula