How to Remove Duplicate Lines: Online, Excel, Google Sheets, Notepad++ and Terminal

The fastest way to remove duplicate lines is to paste the text into Remove Duplicate Lines. The tool keeps the first copy of each line and deletes the rest, so the remaining lines stay in their original order. If the list already lives in a spreadsheet, use Excel’s or Google Sheets’ Remove Duplicates command. In a text file, use Notepad++ or one line in a terminal. Whichever method you use, decide two things first: should “Apple” and “apple” count as the same line, and should a stray space at the end of a line make it different? Each method answers those questions differently, and the differences are listed below.

Remove duplicate lines online in four steps

1. Paste your list into Remove Duplicate Lines, one item per line.
2. Choose the options. Case-sensitive (Apple ≠ apple) is off by default. Ignore extra spaces (start, end, doubled, zero-width) is on. Keep is set to First occurrence. Remove empty lines too is off.
3. Read the result. The note under the output shows how many duplicate lines were removed, and Repeated lines lists each repeated line with the number of times it appeared.
4. Copy the cleaned list, or pass it straight to Sort Lines or Line Counter.

This is the tool’s sample list run with the default options:

Before	After (defaults)	After (Case-sensitive on)
apple	apple	apple
banana	banana	banana
Apple	cherry	Apple
cherry	date	cherry
banana␣␣ (two trailing spaces)		date
banana
date

With the defaults, 3 lines are removed: “Apple” matches “apple”, and both later “banana” lines match the first one. With case sensitivity on, “Apple” stays and 2 lines are removed. Turn on Case-sensitive and turn off the spaces option, and only 1 line is removed, because “banana␣␣” with its trailing spaces no longer matches “banana”.

How the tool decides that two lines are duplicates

The whole line is compared, not individual words. These rules come straight from the tool’s code:

• Line breaks. Windows (CRLF), old Mac (CR) and Unix (LF) line endings are all recognized. The output uses plain LF line breaks.
• Letter case. By default, lines are lowercased before they are compared, so [email protected] and [email protected] count as one line. The copy that is kept is printed exactly as you typed it.
• Spaces. With Ignore extra spaces (start, end, doubled, zero-width) on, spaces and tabs at the ends of a line are ignored. Runs of whitespace inside the line count as a single space, so New York (two spaces) and New York are treated as the same line. Non-breaking spaces count as whitespace too, and zero-width spaces (U+200B, U+2060, U+FEFF) are ignored, so a Paris with a hidden zero-width space matches Paris.
• Accents and invisible characters. Lines are compared in Unicode normalized form (NFC), so an é typed as one character and an é typed as e plus a combining accent count as the same line. Accents are not ignored, though: café and cafe are different lines. With the spaces option off, a zero-width space makes a line different. If identical-looking lines survive, run Remove Invisible Characters first.
• Blank lines. Empty lines (including lines with only spaces) are never treated as duplicates. With Remove empty lines too off, they stay where they are, so paragraph spacing survives: a, blank, b, blank, a becomes a, blank, b, blank, and the note says “1 duplicate line removed”. Tick Remove empty lines too to drop every blank line; the note then adds how many empty lines were removed.

First or last occurrence

Keep decides which copy stays. The other lines never move. Take a list of pages a visitor opened, in order:

Input	Keep: First occurrence	Keep: Last occurrence
home	home	about
about	about	pricing
pricing	pricing	home
home	contact	contact
contact

Use First occurrence for most lists. Use Last occurrence when later entries are more current, such as a log where the most recent mention of an item is the one that matters.

Remove duplicates and sort at the same time

If you want the list in alphabetical order anyway, Sort Lines has a Remove duplicates checkbox. It compares lines the way Remove Duplicate Lines does with its spaces option on (ends trimmed, doubled and zero-width spaces ignored) and keeps the first copy. It also follows the tool’s Case-sensitive setting, which is off by default.

Input:                      Sort Lines (Alphabetical, Remove empty lines, Remove duplicates):
pear                        Apple
Apple                       banana
banana                      pear
apple
pear

Banana

For more on sort orders, such as natural order (2 before 10) and ignoring leading quotes or bullets, see how to alphabetize a list.

Excel and Google Sheets

Excel. Select the cells, then choose Data > Data Tools > Remove Duplicates. In the dialog, tick the columns to compare (Select All or Unselect All help) and click OK. A message then tells you how many duplicate values were removed or how many unique values remain. Microsoft documents three points worth knowing:

• The first occurrence of each value is kept.
• Excel compares what appears in the cell, not the underlying value. A date shown as “3/8/2006” and the same date shown as “Mar 8, 2006” are therefore not duplicates.
• Rows are deleted permanently. Microsoft suggests copying the data elsewhere first. Ctrl+Z undoes the change.

To keep the original list and get a deduplicated copy next to it, use the formula =UNIQUE(A2:A100). As of October 2026, Microsoft lists UNIQUE as available in Excel for Microsoft 365, Excel 2024 and Excel 2021, among other versions.

Google Sheets. Select the range, then choose Data > Data cleanup > Remove duplicates. Choose whether the data has a header row and which columns to analyze, then click Remove duplicates. Google states that cells with the same value but different letter case, formatting or formulas count as duplicates. Data > Data cleanup > Trim whitespace removes stray spaces first, but Google notes that it does not trim non-breaking spaces. The formula =UNIQUE(A2:A100) returns rows in the order in which they first appear.

Notepad++

Open the file and go to Edit > Line Operations. There are two commands:

Command	What it removes
Remove Duplicate Lines	Every later copy of a full line, anywhere in the file. Only the first instance is kept.
Remove Consecutive Duplicate Lines	Only copies on the lines directly after the first instance.

Both commands work on the selected lines, or on the whole file if nothing is selected. The manual notes that Remove Duplicate Lines also clears all bookmarks, and that both commands assume the file uses the same line ending throughout.

Command line: sort, uniq, awk and PowerShell

Results below are for a file with the lines apple, banana, Apple, cherry, banana, date:

Command	Output	Keeps order?
uniq list.txt	nothing removed (no two identical lines are adjacent)	yes
sort list.txt | uniq	apple, Apple, banana, cherry, date	no, sorted
sort -u list.txt	same five lines, sorted	no, sorted
sort -fu list.txt	apple, banana, cherry, date	no, sorted
awk '!seen[$0]++' list.txt	apple, banana, Apple, cherry, date	yes
awk '!seen[tolower($0)]++' list.txt	apple, banana, cherry, date	yes

• uniq only removes repeated lines that are adjacent. The GNU manual says to sort the input first, or to use sort -u. uniq -c adds a count to each line, and uniq -d prints only the repeated lines.
• sort -u outputs the first of each run of lines that compare equal. With -f (ignore case), “b” and “B” compare equal and only one is kept. The order of uppercase and lowercase lines depends on your locale.
• awk prints a line only the first time it sees it. This keeps the original order, which the GNU Awk guide recommends when order matters, for example in a shell history file.
• PowerShell: Get-Content list.txt | Sort-Object -Unique. Microsoft’s documentation says that Unique ignores case and that the output is sorted.

Duplicate lines vs. duplicate words

Everything above works on whole lines. To fix “the the” inside a sentence, use Remove Duplicate Words instead. Its default mode, Repeated words in a row (the the), removes only repeated words that sit next to each other on the same line. Every repeat of a word removes later copies anywhere in the text:

Before	Repeated words in a row	Every repeat of a word
Paris in the the spring is is lovely.	Paris in the spring is lovely.	Paris in the spring is lovely.
red blue red green blue	red blue red green blue	red blue green

To check what a cleanup changed, paste the before and after versions into Text Compare.