Computing Text Document Similarity Using TF-IDF and Cosine Similarity

Fundamentals of Text Similarity Computation

In the field of natural language processing, calculating similarity between text documents is a fundamental and important task. TF-IDF (Term Frequency-Inverse Document Frequency) is a widely used text representation method that transforms texts into numerical vectors, enabling subsequent mathematical computations.

TF-IDF Vectorization Process

TF-IDF evaluates word importance by considering both the frequency of words within individual documents and their distribution across the corpus. Term Frequency (TF) measures word importance within a single document, while Inverse Document Frequency (IDF) reduces the weight of common words and enhances the weight of rare words.

In Python, this can be implemented using scikit-learn's TfidfVectorizer:

from sklearn.feature_extraction.text import TfidfVectorizer

corpus = ["I'd like an apple", 
          "An apple a day keeps the doctor away", 
          "Never compare an apple to an orange", 
          "I prefer scikit-learn to Orange", 
          "The scikit-learn docs are Orange and Blue"]

vectorizer = TfidfVectorizer(min_df=1, stop_words="english")
tfidf_matrix = vectorizer.fit_transform(corpus)

Cosine Similarity Calculation

Cosine similarity measures the similarity between two vectors by calculating the cosine of the angle between them. In TF-IDF vector space, this method effectively handles documents of different lengths.

Computing the similarity matrix for all documents:

pairwise_similarity = tfidf_matrix * tfidf_matrix.T

The resulting similarity matrix is a sparse matrix that can be converted to a dense array for inspection:

similarity_array = pairwise_similarity.toarray()
print(similarity_array)

Result Interpretation and Document Retrieval

The diagonal elements of the similarity matrix represent self-similarity and are always 1. To find documents most similar to a specific document, diagonal elements must be ignored:

import numpy as np

# Set diagonal elements to NaN
arr = similarity_array.copy()
np.fill_diagonal(arr, np.nan)

# Find the document most similar to document 4
input_idx = 4
most_similar_idx = np.nanargmax(arr[input_idx])
most_similar_doc = corpus[most_similar_idx]

Importance of Text Preprocessing

Appropriate text preprocessing is essential for improving the accuracy of similarity calculations. Common preprocessing steps include:

import nltk
import string
from nltk.stem import PorterStemmer

def preprocess_text(text):
    # Convert to lowercase
    text = text.lower()
    
    # Remove punctuation
    text = text.translate(str.maketrans('', '', string.punctuation))
    
    # Tokenization
    tokens = nltk.word_tokenize(text)
    
    # Stemming
    stemmer = PorterStemmer()
    stemmed_tokens = [stemmer.stem(token) for token in tokens]
    
    return ' '.join(stemmed_tokens)

Practical Application Considerations

When dealing with large document collections, using sparse matrices can significantly save memory. scikit-learn's TfidfVectorizer returns sparse matrices by default, making it suitable for high-dimensional feature spaces.

For online document comparison tools like Copyleaks and other commercial solutions, more complex similarity detection algorithms are typically integrated, including AI-based semantic similarity computation and multi-language support. These tools can handle multiple file formats and provide detailed similarity reports.

Performance Optimization Recommendations

In practical applications, consider the following optimization strategies:

• Use stop word lists to remove common but meaningless words
• Set appropriate min_df and max_df parameters to filter rare and overly common words
• Consider using n-gram features to capture phrase-level similarity
• Use incremental learning or distributed computing for large-scale corpora

Conclusion

TF-IDF combined with cosine similarity provides a robust and practical framework for text similarity computation. With proper preprocessing and parameter tuning, this approach can deliver accurate similarity assessments across various application scenarios. Understanding the advanced features of commercial tools also helps in making better technology selection decisions for real-world projects.