Remove Duplicate Words

How to use it

1. Paste or type your text in the box, or use Open file for a plain-text file. Example loads a short sample that shows both kinds of repeat.
2. Under Remove, choose Repeated words in a row (the the) to fix typing mistakes, or Every repeat of a word to keep only the first use of each word.
3. Tick Case-sensitive if Apple and apple should count as different words.
4. Read the result and the note above it, for example 2 words removed. Then Copy, Download .txt, or Use as input to run another cleaning step on the result.

The two modes

The sample shows the difference. With the default mode:

Before:
Paris in the the spring is is lovely.
red blue red green blue

After (2 words removed):
Paris in the spring is lovely.
red blue red green blue

The second line is untouched because no word follows itself there. With Every repeat of a word the first line is cleaned the same way and the second becomes red blue green (4 words removed in total).

How words are compared

In a row. A word here is a run of letters, digits and apostrophes, so don’t, it’s, 3 and café each count as one word. The repeat must be on the same line, separated only by spaces or tabs. Anything else between the two words breaks the match: a line break, a comma, a period or a dash. A word joined to another by a hyphen is not compared on its own, so well-known known fact and go go-go go stay unchanged.

Grammatical doubles such as that that or had had are removed too, because the tool cannot tell a typo from a correct sentence. Read the result when you clean prose.

Every repeat. Here the text is split at spaces, tabs and line breaks, and each piece is compared without the punctuation at its start and end. Hello., Hello! and hello? are the same word, and so are (note) and note. Hyphenated words, numbers with commas and email addresses are compared as whole pieces, so well-known is not the same as known.

Edge cases worth knowing

• Case. Without Case-sensitive, Hello hello HELLO becomes Hello: the first spelling stays. With it ticked, nothing is removed.
• Accents. café and cafe are different words, and so are Straße and STRASSE. To treat accented and plain letters as the same, run remove accents first.
• Line breaks in the default mode. Only repeats on the same line count, so line breaks and blank lines are never removed. the at the end of one line followed by the end on the next is left as it is, and x x, a blank line, then x y becomes x, a blank line, x y. To catch a repeat split across lines, join the lines first with remove line breaks.
• Punctuation in the every-repeat mode. Commas and periods after a removed word move to the word before it if that word ends in a letter or digit (buy shoes, cheap shoes, gives buy shoes, cheap,). A removed word that starts with a bracket or quote goes together with its closing half, and only sentence punctuation (. , ; : ! ? …) after it is kept: note (note) gives note, tags: red (red) blue gives tags: red blue and word (word). gives word..
• Lines in the every-repeat mode. The comparison covers the whole text, not each line. When every word on a line is removed, an empty line stays in its place: one, two, one on three lines gives one, two and a blank third line. Remove empty lines deletes it if you don’t need the rows to line up.
• Emoji and symbols. A piece with no letter or digit, such as 😀 or a lone dash, is never removed.
• Long text. Everything runs in your browser. A test text of 200,000 words was processed in well under a second.

When people need it

• Proofreading. Doubled words such as the the or is is are easy to miss on screen, especially across a line break. The default mode only finds repeats on the same line, so for hard-wrapped text remove the line breaks first. Run the default mode, then compare the word count with the word counter.
• Keyword and tag lists. SEO and social media lists built from several sources often repeat terms. Every repeat of a word leaves one copy of each, in the original order.
• Text pasted from PDFs or transcripts. Copy errors and speech-to-text output often repeat a word. Run the default mode after removing line breaks.
• Vocabulary lists. To find out which words repeat before removing them, the word frequency counter lists each word with its count.

Duplicate words vs duplicate lines

This tool compares single words. If your list has one item per line and the items have several words, such as blue shoes and red shoes, Every repeat of a word would remove the second shoes and break the list. Use remove duplicate lines instead: it compares whole lines, so red shoes stays and only exact repeated lines go.

Questions