Remove Duplicate Lines

How to remove duplicate lines

1. Paste your list into Your list (one item per line), or use Open file for a .txt or .csv file. Example loads a short sample.
2. Check the matching options. By default, case and extra spaces are ignored, so Apple and *apple * count as the same line.
3. Choose which copy to keep under Keep: First occurrence (default) or Last occurrence.
4. Read the result. The note shows how many duplicate lines were removed, and Repeated lines lists each line that had copies.
5. Click Copy or Download .txt, or continue with Sort lines, Remove empty lines or Count lines.

The result updates as you type or change an option, and Use as input moves the result into the input box if you want to run another tool on it.

The options and what they change

The options only decide which lines count as duplicates. The line that stays is copied exactly as you typed it: if the first copy is banana␣␣ with two trailing spaces, those spaces stay in the result. To clean the spacing itself, run the result through remove extra spaces.

How duplicates are compared

The tool reads your text line by line. A line ends at a line break of any kind (Windows CRLF, Unix LF or old Mac CR), and the result always uses plain LF line breaks. For each line it builds a comparison key from the options above and keeps a line only if that key has not been seen before. This is why the order of your list does not matter: two copies are found even if they are thousands of lines apart.

The rules that decide the cases people ask about, with default settings:

Two cases can surprise you:

• Blank lines. With Remove empty lines too off, blank lines are never treated as duplicates: every one stays in its place, and the note counts only duplicate lines with text. a, blank, b, blank, a becomes a, blank, b, blank, with “1 duplicate line removed”. When a removed line sat between two blank lines, those blank lines end up next to each other; remove empty lines can collapse them to one.
• Characters that look the same. Accented letters are normalized before comparing, and with the spacing option on, a zero-width space inside a line is ignored. Letters from different alphabets are still different characters, though: a Cyrillic а does not match a Latin a. Remove invisible characters helps when a line still refuses to match.

Worked examples

The example list, with default settings:

apple
banana
Apple
cherry
banana  
banana
date

Repeated lines shows banana 3 times and apple 2 times.

Keeping the last copy is useful when a list is a log and the latest entry matters. Here each line is a status update, in order:

Both keep three lines; Keep last orders them by when each value was last seen.

A keyword list exported from two tools, with default settings:

running shoes
Running Shoes
trail running shoes
running  shoes
waterproof hiking boots
trail running shoes 

becomes running shoes, trail running shoes, waterproof hiking boots, with 3 lines removed.

When people need it

• Mailing lists. Contacts exported from a CRM, a webinar tool and a spreadsheet overlap. Paste the email column and keep the defaults: different capitalization and trailing spaces from the export are ignored.
• Keyword and hashtag lists. Merged keyword research usually repeats the same phrase in different case. Dedupe it, then check keyword density in your draft.
• URLs and file names. Lists of links from a crawl or a folder listing. Turn on Case-sensitive if the server treats /Page and /page as different addresses.
• Code and data. IDs for an SQL IN list, values for a dropdown or entries in a config file. After deduping, the comma separator turns the column into 'a', 'b', 'c'.
• Names and guest lists. Sign-ups collected from several forms. Exact matching cannot catch Jon Smith and John Smith, so read the result once.

How this compares with Excel and the command line

In Excel, Data > Remove Duplicates deletes repeated rows from the selected range, and you choose which columns to compare. Microsoft notes that the duplicates are permanently deleted, so it suggests copying the data first. Here the original stays in the input box.

On Linux and macOS, uniq removes a repeated line only when the copies are next to each other, according to the GNU Coreutils manual, so it is normally used after sort or replaced by sort -u. Both sort the output. This tool finds copies anywhere in the list and keeps your original order, which matters for ranked lists, logs and anything written in a meaningful sequence. If you do want an alphabetical result, the sort lines tool has its own Remove duplicates option. The guide to removing duplicate lines covers more programs.

Questions