FileKiln

Remove Duplicate Lines

Delete duplicate lines from any list, keeping first occurrences in order, or sort and dedupe in one pass. Runs in your browser.

Ready when you arededupe lines
Runs on your computer. Nothing gets uploaded.

Your stuff stays yours. The conversion happens in this page's JavaScript. Nothing you paste here leaves your machine, and the tool keeps working with the network unplugged.

Two honest modes. Remove duplicates keeps the first occurrence of each line and preserves your original order - the right behavior for config files, playlists, and anything where order carries meaning. Sort + dedupe is the classic sort -u: alphabetical order, one of each. Comparison is exact and case-sensitive, so Apple and apple are different lines - deduplication that silently case-folds or trims whitespace destroys data you didn't know you had. Email lists, log extracts, and export files are the usual customers, and none of them belong on someone else's server.

How to remove duplicate lines

  1. Paste your list into Lines in, one entry per line. A column copied from a spreadsheet works, since each cell arrives on its own line.
  2. Choose a mode:
    • Press Remove duplicates to keep the list in its current order and drop every repeat after the first.
    • Press Sort + dedupe to drop repeats and then sort what remains alphabetically.
  3. Check Unique lines out. There is no count shown, so if you need to know how many lines were removed, compare line counts before and after with the word counter.
  4. Press Copy result to take the cleaned list, or Clear to start over.

What counts as a duplicate, and how sorting works

The tool splits your text at every line break and compares whole lines as exact strings. Two lines are duplicates only when every character matches, including capitalization, leading and trailing spaces, tabs, and look-alike characters such as a non-breaking space or a curly apostrophe. Nothing is trimmed or normalized first.

Blank lines are lines too. The first empty line in your text is kept and every later empty line is removed as a duplicate of it. In a list with blank lines between groups, the result is that only the first gap survives.

Sort + dedupe uses the browser's locale-aware string comparison rather than raw character codes. In practice, for English text that means:

  • Letters sort alphabetically regardless of case, so apple, Apple, and banana end up next to each other in that order instead of all capitals first. The exact placement of apple versus Apple is decided by your browser's locale rules.
  • Accented letters sort near their base letter, so éclair lands among the e words rather than after z.
  • Numbers inside text are compared digit by digit, not by value. item10 sorts before item2 because the character 1 comes before 2.
  • An empty line sorts before everything else, so it appears at the very top of the output.

A worked example

Here is a messy list with a case variant, a blank line, a repeat, a line with a trailing space after cherry, and two numbered items:

banana
Apple
apple

cherry
banana
cherry
item10
item2

Remove duplicates returns:

banana
Apple
apple

cherry
cherry
item10
item2

Only the second banana was removed. Apple and apple differ by case, and the two cherry lines differ by the trailing space, so all of them stay. Sort + dedupe on the same input returns:


apple
Apple
banana
cherry
cherry
item10
item2

The first line of that output is the blank line, moved to the top by the sort.

Common problems and how to fix them

Obvious duplicates are still there

Invisible differences are the cause almost every time: a trailing space, a tab instead of a space, or a non-breaking space from a web page or spreadsheet. Clean the whitespace in a text editor with a find-and-replace, then run the tool again. Checking the two lines in the text diff checker will confirm whether they really differ.

Email addresses differ only by capitalization

The domain part of an email address is not case-sensitive, and many mail providers ignore case in the part before the @ as well, but this tool compares exactly. Lowercase the whole list first (most spreadsheets have a LOWER function), then dedupe.

Numbered lines sort in the wrong order

Because sorting compares characters, 10 comes before 9. Pad numbers with leading zeros (item02, item10) before sorting, or use Remove duplicates and keep an order you already controlled.

A blank line appears at the top after sorting

Your input contained at least one empty line, often a trailing line break at the end of a pasted list. Delete the first line of the output, or remove blank lines from the input before pressing Sort + dedupe.

Rows from a CSV file are not treated as duplicates

The whole row is compared, so two records with the same name but different dates are different lines. If you want to dedupe on one column, copy only that column into the tool.

Questions this tool gets asked

Does it treat “Apple” and “apple” as duplicates?

No - comparison is exact, including case and whitespace. That is deliberate: silent case-folding corrupts data like IDs and passwords. Run the text through the case converter first if you want case-insensitive deduplication.

Which copy of a duplicate is kept?

The first one, in both modes. In order-preserving mode everything else about your ordering is untouched.

Do trailing spaces make lines “different”?

Yes, because the bytes differ - “cherry” and “cherry ” are distinct lines. If your data may carry stray trailing whitespace, clean it first; invisible whitespace is the most common reason two “identical” lines both survive.

How big a list can it handle?

Hundreds of thousands of lines comfortably - the dedupe is a single pass over the text inside your browser.

Related tools