Question 1

What algorithm does this tool use to remove duplicates?

Accepted Answer

The tool uses a single-pass Set-based deduplication algorithm. It iterates over each line once, building a normalised comparison key (after optional trimming and case folding), and uses a JavaScript Set to track which keys have already been seen. Lines whose key is already in the Set are marked as duplicates and excluded from the output. This runs in O(n) time relative to the number of lines.

Question 2

Is my data safe when I use this tool?

Accepted Answer

Yes, completely. The remove duplicate lines tool runs 100% client-side in your browser using JavaScript. No text is ever sent to a server, stored in a database, or logged anywhere. It is safe to paste email lists, passwords, confidential documents, or any other sensitive data — none of it leaves your device.

Question 3

What is the difference between Case Sensitive and Case Insensitive mode?

Accepted Answer

In case-insensitive mode (the default), the tool converts each line to lowercase before comparing, so "Apple", "APPLE", and "apple" are all treated as the same entry and only the first is kept. In case-sensitive mode, those three lines are considered distinct and all three are preserved. Use case-sensitive mode when you are working with data where capitalisation is meaningful — for example, programming identifiers or passwords.

Question 4

Why would I want to Keep Last Occurrence instead of first?

Accepted Answer

In some workflows, the last occurrence of a value is the most up-to-date version. For example, if you have a log file where each line represents an event and duplicate IDs appear because an entry was updated, keeping the last line gives you the most recent state. The default (keep first) is better when you want to preserve the original order of first appearances.

Question 5

What does the Trim Whitespace option do exactly?

Accepted Answer

When Trim Whitespace is enabled, the tool strips leading and trailing space characters from each line before comparing it to others. This means a line that reads "  hello  " (with surrounding spaces) is compared as "hello" and will match a line that reads "hello" exactly. The original line content — including its spaces — is preserved in the output; only the comparison key is trimmed, not the line itself.

Question 6

Can I use this tool to deduplicate CSV or spreadsheet data?

Accepted Answer

Yes, but the tool operates on whole lines. If you paste one value per line (a single column from a spreadsheet), it works perfectly. For multi-column CSV data, each full row is compared as a unit — so two rows are only considered duplicates if every field matches. To deduplicate by a single column in multi-column data, extract that column first, deduplicate it here, then rejoin. Need to compare two versions of a list? Try our compare-text tool.

Question 7

What happens to blank lines?

Accepted Answer

Blank lines are treated like any other line. If your input has multiple blank lines, all but the first (or last, depending on your setting) will be removed as duplicates. If you want to remove all blank lines entirely, use the Remove Spaces tool which has a dedicated Remove Empty Lines option. Combining both tools gives you a fully cleaned list.

Question 8

Does removing duplicates change the order of my remaining lines?

Accepted Answer

No. This tool preserves your original order, which is the behaviour most people actually want and the one many alternatives get wrong. The common shortcut elsewhere is to sort the list first, because sorting puts identical lines next to each other and makes deduplication trivial — but it also destroys whatever sequence your data was in. That matters for a changelog, a ranked list, or a CSV whose row order carries meaning. Here, the first occurrence of each line stays exactly where it was (or the last, if you switch the Keep Last Occurrence option), and everything else shifts up to close the gap. If you do want the result sorted, run Sort Lines afterwards as a separate, deliberate step.

Question 9

Why do identical-looking lines survive deduplication?

Accepted Answer

Because they are not byte-identical. The three usual causes are a trailing space on one of them, a non-breaking space (U+00A0) that was pasted in where a normal space belongs, and Windows CRLF line endings mixed with Unix LF. None of these are visible on screen, and all three make two lines genuinely different strings. Turn on Trim Whitespace, which removes leading and trailing spaces before comparing — that resolves the majority of cases. If lines still survive, the difference is inside the line rather than at its edges, and running the text through Remove Spaces first will normalise the invisible characters.

Remove Duplicate Lines

Why Use This Tool?

How to Use It

Frequently Asked Questions

Further reading

Why Use This Tool?

How to Use It

Frequently Asked Questions

Further reading

Explore More Free Text Tools