Deep Analysis and Practice of Property-Based Distinct in Java 8 Stream Processing

Introduction

In the Stream API introduced in Java 8, the distinct() method provides a convenient way to remove duplicate elements from a stream. However, this method relies on the implementation of the object's equals() and hashCode() methods, which proves insufficiently flexible in many practical scenarios. When we need to perform distinct operations based on specific properties of objects (such as name, ID, etc.), the standard distinct() method cannot meet these requirements.

Problem Analysis

Consider a collection containing Person objects, where each Person object has a name property. If we wish to perform distinct operations based on names, preserving the first occurrence of each name, the standard distinct() method cannot directly achieve this unless we override the equals() and hashCode() methods of the Person class, which may be infeasible or inappropriate in certain situations.

Core Solution

By creating a custom Predicate function, we can implement distinct functionality based on arbitrary properties. Here is a generic implementation of the distinctByKey method:

public static <T> Predicate<T> distinctByKey(Function<? super T, ?> keyExtractor) {
    Set<Object> seen = ConcurrentHashMap.newKeySet();
    return t -> seen.add(keyExtractor.apply(t));
}

The working principle of this method is as follows:

• Uses ConcurrentHashMap.newKeySet() to create a thread-safe collection for recording seen key values
• Extracts key properties from objects through the keyExtractor function
• Utilizes the return value of Set.add() method (returns true if added successfully, false if already exists) to filter duplicates

Usage Example

Application for distinct operations on a list of Person objects:

List<Person> distinctPersons = persons.stream()
    .filter(distinctByKey(Person::getName))
    .collect(Collectors.toList());

Concurrency Safety Considerations

The use of ConcurrentHashMap.newKeySet() ensures thread safety in parallel stream environments. When the stream executes in parallel, this method can properly handle concurrent access and avoid data race issues.

Behavioral Characteristics Analysis

In the case of ordered streams executing in parallel, this method preserves an arbitrary instance among duplicates, unlike distinct() which always preserves the first encountered instance. This behavioral difference needs to be considered during design.

Alternative Solutions Comparison

Another common implementation approach uses Collectors.toMap():

Collection<Person> distinctPersons = persons.stream()
    .collect(Collectors.toMap(Person::getName, p -> p, (p, q) -> p))
    .values();

Advantages of this method include:

• For ordered streams, always preserves the first encountered duplicate
• Relatively concise and intuitive code
• Potentially better performance in certain scenarios

However, this method immediately terminates the lazy evaluation characteristic of streams and may incur higher memory overhead when processing large amounts of data.

Extended Application: Multi-Property Distinct

Reference Article 2 provides extended solutions for distinct operations based on multiple properties. We can create a distinctByKeys method that supports variable arguments:

private static <T> Predicate<T> distinctByKeys(final Function<? super T, ?>... keyExtractors) {
    final Map<List<?>, Boolean> seen = new ConcurrentHashMap<>();
    return t -> {
        final List<?> keys = Arrays.stream(keyExtractors)
            .map(ke -> ke.apply(t))
            .collect(Collectors.toList());
        return seen.putIfAbsent(keys, Boolean.TRUE) == null;
    };
}

Usage example:

List<Person> distinctPersons = persons.stream()
    .filter(distinctByKeys(Person::getFirstName, Person::getLastName))
    .collect(Collectors.toList());

Custom Key Class Approach

Another more type-safe method involves using custom key classes:

record PersonKey(String firstName, String lastName) {
    public PersonKey(Person person) {
        this(person.getFirstName(), person.getLastName());
    }
}

public static <T> Predicate<T> distinctByKeyClass(Function<? super T, Object> keyExtractor) {
    Map<Object, Boolean> seen = new ConcurrentHashMap<>();
    return t -> seen.putIfAbsent(keyExtractor.apply(t), Boolean.TRUE) == null;
}

Usage pattern:

List<Person> distinctPersons = persons.stream()
    .filter(distinctByKeyClass(PersonKey::new))
    .collect(Collectors.toList());

Performance Considerations

When selecting a distinct solution, consider the following performance factors:

• Memory usage: Solutions based on ConcurrentHashMap require additional memory to store seen key values
• Computational complexity: All solutions have O(n) complexity, but constant factors may differ
• Parallel performance: ConcurrentHashMap performs well in parallel environments but may experience lock contention
• Stream characteristics: filter operations maintain stream laziness, while collect operations trigger immediate evaluation

Best Practice Recommendations

Based on practical project experience, we recommend the following best practices:

• For simple single-property distinct operations, prioritize using the distinctByKey method
• When distinct operations based on multiple properties are needed, consider using custom key class approaches for better type safety
• In memory-sensitive scenarios, evaluate the memory overhead of different solutions
• In parallel stream environments, always use thread-safe collection implementations
• Consider the ordering requirements of distinct operations and choose appropriate preservation strategies

Conclusion

Although Java 8 Stream API does not directly provide built-in methods for property-based distinct operations, we can flexibly implement this functionality through custom Predicate functions. The distinctByKey method and its variants introduced in this article provide efficient, safe solutions that can meet the requirements of various complex business scenarios. Developers should choose appropriate implementation solutions based on specific performance requirements, memory constraints, and business logic.