JavaScript Array Merging and Deduplication: From Basic Methods to Modern Best Practices

Introduction

Array merging with duplicate removal is a common and essential operation in JavaScript development. Whether processing user input, integrating API data, or optimizing data structures, efficient and reliable array merging and deduplication solutions are required. This article starts from fundamental concepts and progressively explores various implementation methods.

Basic Array Merging Methods

Before discussing deduplication, it's crucial to understand basic array merging approaches. JavaScript provides multiple methods for array merging, each with specific use cases and characteristics.

Using concat Method

The concat method is a built-in method on the JavaScript Array prototype, used to merge two or more arrays, returning a new array without modifying the original arrays.

// ES5 syntax example
var array1 = ["Vijendra", "Singh"];
var array2 = ["Singh", "Shakya"];
var mergedArray = array1.concat(array2);
console.log(mergedArray); // Output: ["Vijendra", "Singh", "Singh", "Shakya"]

This approach is straightforward but doesn't automatically remove duplicates, resulting in an array containing all original elements, including duplicates.

Using Spread Operator

ES6 introduced the spread operator, providing more concise syntax for array merging.

// ES6 syntax example
const array1 = ["Vijendra", "Singh"];
const array2 = ["Singh", "Shakya"];
const mergedArray = [...array1, ...array2];
console.log(mergedArray); // Output: ["Vijendra", "Singh", "Singh", "Shakya"]

The spread operator not only offers clean syntax but also integrates better with modern JavaScript features.

Array Deduplication Implementation Solutions

After completing array merging, duplicate elements need to be removed. Here are several commonly used deduplication implementation approaches.

Double Loop Deduplication Algorithm

This is the most fundamental deduplication method, using nested loops to traverse the array and compare each element for duplicates.

function arrayUnique(array) {
    var result = array.concat();
    for (var i = 0; i < result.length; i++) {
        for (var j = i + 1; j < result.length; j++) {
            if (result[i] === result[j]) {
                result.splice(j, 1);
                j--; // Adjust index to handle array after deletion
            }
        }
    }
    return result;
}

// Usage example
var array1 = ["Vijendra", "Singh"];
var array2 = ["Singh", "Shakya"];
var uniqueArray = arrayUnique(array1.concat(array2));
console.log(uniqueArray); // Output: ["Vijendra", "Singh", "Shakya"]

The advantage of this method is simple implementation without dependency on modern JavaScript features, offering good compatibility. The disadvantage is O(n²) time complexity, making it inefficient for large arrays.

Prototype Method Extension

Array prototype can be extended to create reusable deduplication methods, but care must be taken to avoid polluting the global prototype.

Object.defineProperty(Array.prototype, 'unique', {
    enumerable: false,
    configurable: false,
    writable: false,
    value: function() {
        var result = this.concat();
        for (var i = 0; i < result.length; i++) {
            for (var j = i + 1; j < result.length; j++) {
                if (result[i] === result[j]) {
                    result.splice(j, 1);
                    j--;
                }
            }
        }
        return result;
    }
});

// Usage example
var array1 = ["Vijendra", "Singh"];
var array2 = ["Singh", "Shakya"];
var uniqueArray = array1.concat(array2).unique();
console.log(uniqueArray); // Output: ["Vijendra", "Singh", "Shakya"]

Modern JavaScript Solutions

With the evolution of ECMAScript standards, more elegant and efficient solutions have emerged.

Using Set Data Structure

The Set object introduced in ES6 is a collection of values where each value must be unique, providing a perfect solution for deduplication problems.

// Using spread operator and Set
const array1 = ["Vijendra", "Singh"];
const array2 = ["Singh", "Shakya"];
const uniqueArray = [...new Set([...array1, ...array2])];
console.log(uniqueArray); // Output: ["Vijendra", "Singh", "Shakya"]

// Using concat and Set
const uniqueArray2 = [...new Set(array1.concat(array2))];
console.log(uniqueArray2); // Output: ["Vijendra", "Singh", "Shakya"]

The Set solution offers advantages in code simplicity and excellent performance with O(n) time complexity. However, note that Set uses strict equality comparison, requiring special handling for object references.

Generic Merge Function

For more complex scenarios, a generic merge function can be created, supporting custom comparison predicates.

const mergeArrays = (arrayA, arrayB, predicate = (a, b) => a === b) => {
    const result = [...arrayA];
    arrayB.forEach(itemB => {
        if (!result.some(itemA => predicate(itemA, itemB))) {
            result.push(itemB);
        }
    });
    return result;
};

// Primitive type usage example
const basicResult = mergeArrays(['a', 'b', 'c'], ['c', 'x', 'd']);
console.log(basicResult); // Output: ['a', 'b', 'c', 'x', 'd']

// Object type usage example
const objectResult = mergeArrays(
    [{id: 1}, {id: 2}], 
    [{id: 2}, {id: 3}], 
    (a, b) => a.id === b.id
);
console.log(objectResult); // Output: [{id: 1}, {id: 2}, {id: 3}]

Performance Analysis and Comparison

Different deduplication methods show significant performance differences, requiring careful selection based on data scale and runtime environment.

Time Complexity Analysis

The double loop method has O(n²) time complexity, performing poorly with large arrays. The Set method has O(n) time complexity, offering better scalability. The indexOf method also has O(n²) time complexity since indexOf itself requires array traversal.

Actual Performance Testing

Based on actual test data, the Set method typically shows the best performance:

• Double loop method: approximately 0.4-0.5 milliseconds
• indexOf method: approximately 0.5-0.6 milliseconds
• Set method: approximately 0.1-0.2 milliseconds

Special Scenario Handling

In practical applications, special data types and scenarios may need to be addressed.

Object Array Deduplication

For arrays containing objects, custom comparison logic is required since object reference comparison checks memory addresses rather than content.

const objectArray1 = [{name: "John"}, {name: "Jane"}];
const objectArray2 = [{name: "Jane"}, {name: "Bob"}];

// Using custom comparison function
const uniqueObjects = mergeArrays(
    objectArray1, 
    objectArray2, 
    (a, b) => a.name === b.name
);
console.log(uniqueObjects); // Output: [{name: "John"}, {name: "Jane"}, {name: "Bob"}]

Maintaining Element Order

Some application scenarios require maintaining the original element order. The Set method naturally preserves insertion order, while other methods require additional processing.

Best Practice Recommendations

Based on comprehensive consideration of performance, readability, and maintainability, the following best practices are recommended:

Modern Environment Recommendations

In environments supporting ES6, prioritize the Set solution:

const mergeAndDeduplicate = (arr1, arr2) => [...new Set([...arr1, ...arr2])];

Compatibility Considerations

For scenarios requiring compatibility with older browsers, use indexOf-based solutions:

function mergeAndDeduplicateLegacy(arr1, arr2) {
    const merged = arr1.concat(arr2);
    const result = [];
    for (let i = 0; i < merged.length; i++) {
        if (result.indexOf(merged[i]) === -1) {
            result.push(merged[i]);
        }
    }
    return result;
}

Conclusion

JavaScript array merging and deduplication is a seemingly simple problem rich with technical details. From traditional loop-based traversal to modern Set data structures, each method has its applicable scenarios. In actual development, appropriate solutions should be selected based on project requirements, performance needs, and environmental compatibility. For modern JavaScript projects, the Set solution is recommended for optimal performance and code simplicity; for scenarios requiring complex object handling or special comparison logic, custom merge functions provide maximum flexibility.