Remove Duplicates from Array Java: A Complete Guide

Arrays are fundamental data structures in Java, used to store collections of elements of the same type. However, when working with arrays, it's necessary to remove duplicate elements to ensure data integrity or optimize storage and processing. Java provides several approaches to remove duplicates, each with its strengths and weaknesses.

From traditional loops to modern data structures like HashSet and Stream API, developers have multiple options to eliminate duplicate values. The choice of method depends on factors such as performance requirements, array size, and whether the array is sorted or unsorted.

There are multiple ways to remove duplicates from array in Java. The choice of method depends on factors such as whether the array is sorted or unsorted, performance considerations, and whether we need to preserve the original order of elements.

One simple method to remove duplicates while preserving the original order is by using an ArrayList. This approach involves iterating through the array and adding each element to the ArrayList only if it doesn't already exist.

Steps to Implement

1. Create an ArrayList to store unique elements.
2. Iterate through the array.
3. Add elements to the ArrayList if they are not already present.
4. Convert the ArrayList back to an array.

Example Code to Remove Duplicates from Array in Java Using ArrayList

import java.util.ArrayList;
import java.util.List;

public class RemoveDuplicates {
    public static int[] removeDuplicatesWithList(int[] data) {
        List<Integer> uniqueElements = new ArrayList<>();
        
        for (int value : data) {
            if (!uniqueElements.contains(value)) {  // Check for uniqueness
                uniqueElements.add(value);
            }
        }
        
        return uniqueElements.stream().mapToInt(i -> i).toArray();  // Convert back to array
    }

    public static void main(String[] args) {
        int[] data = {1, 2, 3, 2, 4, 1, 5};
        int[] uniqueData = removeDuplicatesWithList(data);

        for (int num : uniqueData) {
            System.out.print(num + " ");
        }
    }
}

Explanation of the Code

The program removes duplicates from an integer array using an ArrayList. It iterates through the given array and adds each element to the ArrayList only if it is not already present. Since ArrayList.contains(value) checks for existing elements in O(n) time, this approach results in an overall time complexity of O(n²), making it inefficient for large arrays.

After collecting unique elements, the program converts the ArrayList into an integer array using Java Streams (stream().mapToInt(i -> i).toArray()). The main method initializes an integer array {1, 2, 3, 2, 4, 1, 5}, removes duplicates using removeDuplicatesWithList(), and prints the resulting array, maintaining the original order of elements.

Output

1 2 3 4 5 

Pros

• Simple implementation.
• Preserves the original order.
• Easy to understand and implement without additional data structures.
• Works well for small datasets where performance is not a major concern.

Cons

• Inefficient for large arrays due to O(n²) complexity from the contains() method.
• Slower compared to HashSet or LinkedHashSet for large datasets.
• Consumes extra memory due to the use of an ArrayList.
• Not the best choice when performance optimization is required.

LinkedHashSet is part of Java's java.util package and is a hybrid data structure that combines the properties of a HashSet and a LinkedList. It automatically removes duplicate values while maintaining the insertion order of elements, making it a better alternative to HashSet, which does not guarantee order.

Steps to Implement

1. Create a LinkedHashSet – This will store unique elements while preserving their order.
2. Iterate through the array and add elements to LinkedHashSet – Since LinkedHashSet does not allow duplicates, repeated elements will automatically be ignored.
3. Convert the LinkedHashSet back to an array – Use Java Streams to transform the LinkedHashSet into an int[] array.

Example Code to Remove Duplicates from Array in Java Using LinkedHashSet

import java.util.LinkedHashSet;
import java.util.Set;

public class RemoveDuplicates {
    public static int[] removeDuplicatesWithSet(int[] data) {
        Set<Integer> uniqueElements = new LinkedHashSet<>(); // Maintains order while removing duplicates
        
        for (int value : data) {
            uniqueElements.add(value);  // Only unique elements get stored
        }
        
        return uniqueElements.stream().mapToInt(i -> i).toArray();  // Convert back to an int array
    }

    public static void main(String[] args) {
        int[] data = {1, 2, 3, 2, 4, 1, 5};
        int[] uniqueData = removeDuplicatesWithSet(data);

        for (int num : uniqueData) {
            System.out.print(num + " ");
        }
    }
}

Output

1 2 3 4 5 

Explanation of the Code

• A LinkedHashSet<Integer> is created to store unique elements.
• The loop iterates through the input array {1, 2, 3, 2, 4, 1, 5} and adds each element to the LinkedHashSet. Since LinkedHashSet automatically ignores duplicates, only the first occurrences are stored.
• After processing, the LinkedHashSet contains {1, 2, 3, 4, 5} in the same order as they appeared in the array.
• The method then converts the LinkedHashSet back into an integer array using Java Streams (stream().mapToInt(i -> i).toArray()).
• Finally, the array is printed, showing unique elements in the order they were first encountered.

When to Use LinkedHashSet?

• When order matters and you need to remove duplicates efficiently.
• When you want a better-performing alternative to ArrayList.contains() for duplicate checks.
• When you can afford the extra memory usage in exchange for O(n) time complexity.

This approach is a great balance between efficiency and maintaining order, making it suitable for most real-world applications where duplicates need to be removed from lists while keeping the sequence intact.

Pros

• O(n) time complexity for adding elements, making it efficient for large arrays.
• Preserves insertion order, ensuring elements remain in their original sequence.
• Eliminates duplicates automatically while iterating through the array.
• Simple to implement and requires minimal code changes for use.

Cons

• Requires extra memory due to the underlying hash table and linked list.
• Slower than HashSet if only removing duplicates without order is needed.
• Higher space complexity compared to other collections like ArrayList.
• Not as efficient for primitive types as autoboxing adds overhead.

The Stream API, introduced in Java 8, offers a functional approach to Remove Duplicates from Array Java. It provides a clean and concise way to perform various operations, including filtering out duplicates from arrays. Using the distinct() method in the Stream API can efficiently remove duplicates, while retaining the order of elements.

Steps to Implement

1. Create a Stream from the Array: Use Arrays.stream(data) to convert the input array into a stream of elements.
2. Apply distinct() to Remove Duplicates: The distinct() method automatically filters out duplicates by comparing elements using their equals() method.
3. Convert the Stream Back to an Array: The toArray() method is used to collect the distinct elements back into an array.

Example Code to Remove Duplicates from Array in Java Using Java Stream API

import java.util.Arrays;

public class RemoveDuplicates {
    public static int[] removeDuplicatesWithStreams(int[] data) {
        return Arrays.stream(data)      // Convert the array to a stream
                     .distinct()         // Remove duplicates
                     .toArray();        // Convert back to array
    }

    public static void main(String[] args) {
        int[] data = {1, 2, 3, 2, 4, 1, 5};
        int[] uniqueData = removeDuplicatesWithStreams(data);

        for (int num : uniqueData) {
            System.out.print(num + " ");
        }
    }
}

Explanation of the Code

• Arrays.stream(data): Converts the input array into a stream of integers.
• distinct(): This method filters out duplicate values in the stream, leaving only unique elements. The order of elements is preserved since streams maintain the original order unless explicitly modified.
• toArray(): After applying distinct(), the stream is converted back to an array containing only the unique elements.
• In the main method, the input array {1, 2, 3, 2, 4, 1, 5} is processed and printed with duplicates removed. The result is {1, 2, 3, 4, 5}.

Output

1 2 3 4 5 

Pros

• Concise and Readable: The Stream API offers a clean, functional approach to removing duplicates in a single line of code, improving readability and reducing boilerplate code.
• Preserves Order: Like LinkedHashSet, the distinct() method preserves the order of elements as they appear in the original array.
• Functional Style: The Stream API embraces a functional programming paradigm, making the code more declarative and expressive.
• Easy to Use: With minimal setup, the Stream API can be used to perform a variety of data manipulations, including removing duplicates.

Cons

• Performance Varies Based on Implementation: While the Stream API is efficient, the actual performance may depend on the underlying implementation and the size of the dataset. In some cases, it may not perform as well as specialized data structures like LinkedHashSet.
• Higher Overhead for Small Arrays: The overhead of using streams may outweigh the benefits for small arrays, making it less efficient for small datasets.
• No Parallelism by Default: While the Stream API can be parallelized, the default sequential stream may not take full advantage of multi-core processors unless explicitly configured for parallelism, which can limit performance in large-scale data.

When to Use the Java Stream API for Removing Duplicates?

• Concise Code: When you want to write clear and compact code.
• Small to Medium-Sized Datasets: The Stream API is most effective for smaller datasets where readability and ease of use outweigh raw performance.
• Functional Programming Style: If you are adopting a functional programming paradigm and prefer immutable collections or operations.
• Preserving Order: If maintaining the original order of elements is important, and you want to do so without explicitly managing it.

The Stream API is a great choice for many common scenarios, especially when simplicity and readability are prioritized over extreme performance optimizations.

A HashMap is a data structure in Java that stores key-value pairs, where each key is unique. You can use a HashMap to remove duplicates from array Java efficiently by storing the elements as keys in the map. Since keys in a HashMap are unique, any duplicate elements will be ignored when added to the map. This approach is both time-efficient and space-efficient in most cases.

Steps to Implement

1. Create a HashMap: Use a HashMap where the key will be the array element and the value will be a placeholder (true in this case). The value is irrelevant because we only care about the uniqueness of the keys.
2. Add elements to the HashMap: Iterate over the input array, and for each element, add it to the HashMap using the put() method. The HashMap ensures that only unique keys are stored.
3. Retrieve unique elements: After processing the array, you can use keySet() to get the unique keys and convert them into an array.
4. Return the result: Convert the set of unique keys into an array using Java Streams.

Example Code to Remove Duplicates from Array in Java Using HashMap

import java.util.HashMap;
import java.util.Map;

public class RemoveDuplicates {
    public static int[] removeDuplicatesWithMap(int[] data) {
        Map<Integer, Boolean> uniqueElements = new HashMap<>();
        
        // Add elements to the HashMap, only unique keys will be stored
        for (int value : data) {
            uniqueElements.put(value, true);  // The value is irrelevant, only the key is stored
        }
        
        // Convert the keySet to an array and return
        return uniqueElements.keySet().stream().mapToInt(i -> i).toArray();
    }

    public static void main(String[] args) {
        int[] data = {1, 2, 3, 2, 4, 1, 5};
        int[] uniqueData = removeDuplicatesWithMap(data);

        for (int num : uniqueData) {
            System.out.print(num + " ");
        }
    }
}

Explanation of the Code

• HashMap<Integer, Boolean>: A HashMap is created where each unique integer from the array is stored as a key. The value (true) is simply a placeholder because the uniqueness of elements is determined by the keys.
• for (int value: data): The loop iterates through each element in the input array data.
• uniqueElements.put(value, true): The put() method is called to add each element as a key in the HashMap. If the element already exists as a key, it won't be added again.
• uniqueElements.keySet().stream().mapToInt(i -> i).toArray(): The keySet() method returns a set of the unique keys in the map, which are the unique elements from the input array. We then convert the set into an integer array using Java Streams.

Output

1 2 3 4 5

Pros

• O(n) time complexity for efficient element addition.
• Handles large arrays well with fast lookups.
• No need for sorting to remove duplicates.
• Works well with complex objects and primitive data types.

Cons

• Extra memory usage due to hash table storage.
• Does not guarantee element order unless using LinkedHashMap.
• Autoboxing overhead for primitive types.
• Not ideal for small arrays due to the overhead of HashMap.

When dealing with sorted arrays, duplicates can be efficiently removed in place without using extra memory for auxiliary data structures. The main advantage of this approach is that it allows the array to be modified directly, which saves both time and space compared to other methods that require additional collections or arrays.

Steps to Implement

1. Check for Empty Array: If the input array is empty, return 0 as there are no elements to process.
2. Use Two Pointers: One pointer (i) keeps track of the position for the next unique element. The other pointer (j) iterates through the array to check each element.
3. Compare Elements: If the current element (nums[j]) is different from the previous unique element (nums[i]), increment i and move the current element to nums[i].
4. Return the Length: After all unique elements have been moved to the front, return i + 1, which is the count of unique elements in the array.

Example Code to Remove Duplicates from Array in Java Using In-Place Removal

public class RemoveDuplicates {
    public int removeDuplicatesInPlace(int[] nums) {
        // Check if the array is empty
        if (nums.length == 0) return 0;

        int i = 0;  // Pointer for the last unique element

        // Iterate through the array starting from index 1
        for (int j = 1; j < nums.length; j++) {
            if (nums[j] != nums[i]) {  // Found a unique element
                i++;  // Move the unique element pointer
                nums[i] = nums[j];  // Move unique element to the front
            }
        }

        // Return the number of unique elements
        return i + 1;
    }

    public static void main(String[] args) {
        int[] nums = {1, 1, 2, 3, 3, 4, 5, 5};
        RemoveDuplicates remover = new RemoveDuplicates();
        int uniqueLength = remover.removeDuplicatesInPlace(nums);

        // Print the unique elements
        for (int i = 0; i < uniqueLength; i++) {
            System.out.print(nums[i] + " ");
        }
    }
}

Explanation of the Code

• if (nums.length == 0) return 0;: The function starts by checking if the input array is empty. If it is, there are no elements to process, so it returns 0.
• int i = 0: The variable i is initialized to 0 and will track the position of the last unique element in the array.
• for (int j = 1; j < nums.length; j++): The loop starts at index 1 and iterates through the array to compare each element with the last unique element (tracked by i).
• if (nums[j] != nums[i]): Whenever a unique element is found (i.e., the current element is not equal to the previous unique element), we move the pointer i to the next position and place the unique element there.
• return i + 1;: After the loop finishes, the function returns i + 1, which represents the number of unique elements in the array.

Output

1 2 3 4 5

Pros

• Memory efficient as it modifies the array in-place.
• Time efficient with O(n) time complexity.
• Simple to implement using the two-pointer approach.
• Optimal for sorted arrays with no need for extra data structures.

Cons

• Only works for sorted arrays; requires sorting for unsorted ones.
• Modifies the original array, with no original array preservation.
• Not suitable for unsorted arrays; alternative methods needed.
• Fixed array size, trailing elements remain after unique ones.

Remove duplicates from array Java is a common task, and there are several efficient methods available in Java, each with its strengths and limitations. For simple cases, ArrayList and LinkedHashSet provide easy-to-implement solutions, while HashMap is optimal for larger datasets with its O(n) time complexity. 

If memory efficiency is critical, in-place removal is the best choice for sorted arrays, offering O(n) time complexity with no extra space required. The Stream API provides a concise and readable solution, but it can vary in performance. Understanding the array's properties, such as whether it's sorted or unsorted, can help you choose the best method to remove duplicates from Array in Java based on the problem.

1. What is the simplest way to remove duplicates from an array in Java? 

The simplest way is to use an ArrayList, where you iterate through the array and add each element to the list if it doesn't already exist. However, this method can become inefficient for larger arrays due to its O(n²) complexity.

2. What is the best method for removing duplicates from large arrays? 

For large arrays, using a LinkedHashSet or a HashMap is ideal. These methods offer O(n) time complexity, making them highly efficient for large datasets.

3. Can I remove duplicates from an array without using extra memory? 

Yes, for sorted arrays, you can remove duplicates in-place using a two-pointer approach. This method doesn’t require extra space but is only applicable to sorted arrays.

4. What happens if I try to remove duplicates from an unsorted array? 

If the array is unsorted, using methods like LinkedHashSet or HashMap will work well. If you choose the in-place approach, the array must first be sorted, which may add additional time complexity.

5. How does Java’s Stream API help in removing duplicates? 

Java’s Stream API provides a concise, one-line solution for removing duplicates using the distinct() method. It’s highly readable, but performance can vary based on the implementation and array size.

6. Why would I choose a HashMap to remove duplicates? 

A HashMap allows for efficient duplicate removal with O(n) time complexity. It also works well with large datasets, though it consumes more memory than other methods.

7. Does in-place removal work for unsorted arrays? 

No, in-place removal only works for sorted arrays. If the array is unsorted, you will need to either sort it first or use other methods like LinkedHashSet or HashMap for duplicate removal.