Calculating Cosine Similarity with TF-IDF: From String to Document Similarity Analysis

Introduction

In natural language processing (NLP), measuring the similarity between two texts is a fundamental and crucial task. Cosine similarity, as a common metric, evaluates similarity by calculating the cosine of the angle between two vectors in space. Based on the best answer from Stack Overflow Q&A data, this article elaborates on how to implement cosine similarity calculation between two strings without relying on external libraries.

Core Concepts and Formula

The formula for cosine similarity is:

cosine_similarity = (A·B) / (||A|| * ||B||)

where A and B are vector representations of the texts, A·B denotes the dot product, and ||A|| and ||B|| are the magnitudes of the vectors. The value ranges from -1 to 1, with values closer to 1 indicating higher similarity.

Detailed Implementation Steps

Text Preprocessing and Vectorization

First, raw text needs to be converted into vector representations. A simple yet effective approach is to use term frequency vectors. The following code demonstrates how to extract words using regular expressions and generate term frequency vectors:

import re
from collections import Counter

WORD = re.compile(r"\w+")

def text_to_vector(text):
    words = WORD.findall(text)
    return Counter(words)

For example, for the string "This is a foo bar sentence .", the text_to_vector function returns a Counter object containing the occurrence counts of each word: {'This': 1, 'is': 1, 'a': 1, 'foo': 1, 'bar': 1, 'sentence': 1}. Note that punctuation is filtered by the regex, retaining only word characters.

Cosine Similarity Computation

Next, implement the cosine similarity function. This function takes two term frequency vectors, computes their dot product and magnitudes, and applies the formula:

import math

def get_cosine(vec1, vec2):
    intersection = set(vec1.keys()) & set(vec2.keys())
    numerator = sum([vec1[x] * vec2[x] for x in intersection])

    sum1 = sum([vec1[x] ** 2 for x in list(vec1.keys())])
    sum2 = sum([vec2[x] ** 2 for x in list(vec2.keys())])
    denominator = math.sqrt(sum1) * math.sqrt(sum2)

    if not denominator:
        return 0.0
    else:
        return float(numerator) / denominator

Key steps include: computing the set of common words (intersection) between the two vectors for dot product calculation; calculating the sum of squares for each vector's magnitudes; and handling cases where the denominator is zero to avoid division errors.

Example Application

Test with the provided example strings:

s1 = "This is a foo bar sentence ."
s2 = "This sentence is similar to a foo bar sentence ."
s3 = "What is this string ? Totally not related to the other two lines ."

vector1 = text_to_vector(s1)
vector2 = text_to_vector(s2)
vector3 = text_to_vector(s3)

print("Cosine similarity between s1 and s2:", get_cosine(vector1, vector2))
print("Cosine similarity between s1 and s3:", get_cosine(vector1, vector3))
print("Cosine similarity between s2 and s3:", get_cosine(vector2, vector3))

The output should show high similarity between s1 and s2 (approximately 0.86), and low similarity between s1 and s3, and s2 and s3, aligning with intuitive expectations.

Advanced Discussion: TF-IDF Weighting and Semantic Representation

Importance of TF-IDF

The above implementation relies solely on term frequency, without considering word importance across a corpus. TF-IDF (Term Frequency-Inverse Document Frequency) is a common weighting scheme that optimizes similarity calculation by reducing weights of common words and increasing those of rare words. However, as noted in the Q&A data, applying TF-IDF requires a large corpus to estimate IDF values, otherwise it may introduce noise.

Challenges in Semantic Representation

The second answer in the Q&A data highlights limitations of term frequency-based methods: they fail to capture semantic information of words. For instance, "cat" and "dog" might be close in vector space, but simple term frequency vectors cannot reflect this. More advanced methods involve learning distributed representations of words (e.g., Word2Vec, GloVe) and combining these vectors to represent phrases or sentences. However, this faces challenges like data sparsity and compositional semantics, where simple additive models may not distinguish between "cats chase dogs" and "dogs chase cats".

Optimizations and Extensions

To improve the accuracy of similarity calculation, consider the following optimizations:

• Text preprocessing: Incorporate stemming or lemmatization to normalize word forms.
• Stopword removal: Eliminate common but semantically weak words (e.g., "the", "is").
• Using n-grams: Consider word sequences instead of single words to capture local context.

For large-scale applications, it is recommended to use optimized libraries (e.g., scikit-learn's TfidfVectorizer and cosine_similarity functions) for better performance.

Conclusion

This article details a pure Python implementation for calculating cosine similarity between strings, from basic term frequency methods to advanced discussions on TF-IDF and semantic representation. Through practical code examples and theoretical analysis, it demonstrates how to evaluate text similarity and its potential and limitations in real-world applications. For beginners, this serves as a solid starting point; for advanced users, the challenges and extensions mentioned provide avenues for further exploration.