Applying LINQ Distinct Method to Extract Unique Field Values from Object Lists in C#

Fundamental Application of LINQ Distinct Method

In C# programming, when processing object collections, there is often a need to extract unique values of specific fields. LINQ (Language Integrated Query) provides powerful query capabilities, with the Distinct method being the core tool for implementing deduplication functionality.

Consider the following object definition:

public class CustomObject
{
    public int TypeId { get; set; } // 10 types, value range 0-9
    public string UniqueString { get; set; } // unique identifier
}

Assuming there is a list containing 100 elements of CustomObject, but only 10 distinct TypeId values. To extract these unique type IDs, the simplest LINQ query can be used:

List<CustomObject> objList = GetObjectList();
IEnumerable<int> uniqueTypeIds = objList.Select(o => o.TypeId).Distinct();

This approach first uses the Select operator to project objects to the TypeId field, then applies the Distinct method to remove duplicate values. Since the int type implements the default equality comparer, this method can correctly identify duplicate integer values.

Working Principle of Distinct Method

The Distinct method in LINQ to Objects is implemented based on hash sets. When processing value types, such as the int type in the example, the system uses the default equality comparer. For reference types, special attention must be paid to equality definitions.

The method execution process is as follows:

// Simulating internal implementation of Distinct
public static IEnumerable<TSource> DistinctImpl<TSource>(IEnumerable<TSource> source)
{
    HashSet<TSource> seen = new HashSet<TSource>();
    foreach (TSource item in source)
    {
        if (seen.Add(item))
        {
            yield return item;
        }
    }
}

This implementation ensures algorithm time complexity of O(n) and space complexity of O(k), where k is the number of unique elements.

Handling Deduplication Requirements for Complex Objects

When deduplication needs to be based on specific properties of objects rather than entire objects, the situation becomes more complex. For example, to obtain a complete list of objects with unique TypeId, not just the ID values.

Using the GroupBy operator can achieve this requirement:

IEnumerable<CustomObject> distinctObjects = objList
    .GroupBy(o => o.TypeId)
    .Select(g => g.First());

This method first groups objects by TypeId, then selects the first element from each group. While functionally meeting the requirement, it may be less efficient in performance compared to specialized deduplication methods.

Custom DistinctBy Extension Method

To provide a more elegant solution, a custom DistinctBy extension method can be created:

public static IEnumerable<TSource> DistinctBy<TSource, TKey>(
    this IEnumerable<TSource> source, 
    Func<TSource, TKey> keySelector)
{
    if (source == null) throw new ArgumentNullException(nameof(source));
    if (keySelector == null) throw new ArgumentNullException(nameof(keySelector));
    
    HashSet<TKey> seenKeys = new HashSet<TKey>();
    
    foreach (TSource element in source)
    {
        TKey key = keySelector(element);
        if (seenKeys.Add(key))
        {
            yield return element;
        }
    }
}

Usage method:

IEnumerable<CustomObject> distinctByType = objList.DistinctBy(o => o.TypeId);

This implementation provides better performance and code readability, particularly suitable for repeated use in large projects.

Performance Comparison and Best Practices

Different methods show significant performance differences:

• Basic Distinct: Suitable for simple value type deduplication, optimal performance
• GroupBy Method: Powerful but relatively heavy, suitable for complex grouping scenarios
• Custom DistinctBy: Balances performance and flexibility, suitable for property-based deduplication

In actual development, it is recommended:

// Scenario 1: Only field values needed
var typeIds = objList.Select(o => o.TypeId).Distinct().ToList();

// Scenario 2: Complete objects needed (using custom method)
var distinctObjects = objList.DistinctBy(o => o.TypeId).ToList();

// Scenario 3: Multi-field deduplication
var multiDistinct = objList.DistinctBy(o => new { o.TypeId, o.UniqueString });

Advanced Application: Custom Equality Comparer

For complex equality judgments, custom IEqualityComparer can be implemented:

public class TypeIdComparer : IEqualityComparer<CustomObject>
{
    public bool Equals(CustomObject x, CustomObject y)
    {
        if (ReferenceEquals(x, y)) return true;
        if (x is null || y is null) return false;
        return x.TypeId == y.TypeId;
    }
    
    public int GetHashCode(CustomObject obj)
    {
        return obj?.TypeId.GetHashCode() ?? 0;
    }
}

// Using custom comparer
var distinctWithComparer = objList.Distinct(new TypeIdComparer());

This method provides maximum flexibility but requires more code implementation.

Conclusion

LINQ's Distinct method and related technologies provide C# developers with powerful data deduplication capabilities. Choose the appropriate implementation based on specific requirements: for simple field value deduplication, use Select+Distinct combination; for property-based object deduplication, recommend using custom DistinctBy method; when complex equality logic is needed, implementing custom comparers is the best choice.

Understanding the internal implementations and performance characteristics of these methods helps make optimal technical choices in specific scenarios, improving code efficiency and maintainability.