Remove Invisible Characters

How to use it

1. Paste your text into the box, open a plain-text file with Open file, or press Example to load a sample with five hidden characters in it.
2. Leave Replace no-break and other special spaces with a normal space ticked unless you want to keep no-break spaces on purpose.
3. Leave Keep joiners inside emoji (👨‍👩‍👧) ticked if the text has emoji made of several emoji joined together.
4. Read the table under the result: it names each character found and how many times, most frequent first.
5. Press Copy or Download .txt. Use as input lets you run a second clean-up, for example with the extra spaces remover.

The example text looks like an ordinary sentence, but it contains two zero-width spaces, a no-break space, a word joiner and a byte order mark. The tool reports 5 characters removed or replaced and returns This text looks normal but has hidden characters.

Which characters it finds

The tool checks for 40 characters plus the whole block of Unicode tag characters. They fall into four groups, and each group is handled differently.

Normal spaces, tabs and line breaks are never touched. Characters outside this list stay as they are, including variation selectors such as U+FE0F. The space look-alikes are the same set that Remove extra spaces turns into normal spaces. A zero-width non-joiner or zero-width joiner between two letters or marks of a non-Latin script (Persian, Hindi) is kept, whatever the options, because there it is part of the spelling; between Latin letters or anywhere else it is removed. Tag characters are listed in the table as one line, “Tag character (U+E0000–U+E007F)”.

The options in detail

Replace no-break and other special spaces with a normal space. A no-break space looks like a space but stops a line from breaking there, and many programs do not treat it as a space when they search, split or compare text. With the option on, each of the 15 space look-alikes becomes one ordinary space and is counted in the table. With it off, they stay and are not listed, so only the zero-width and control characters are removed.

Each special space becomes exactly one space, so two figure spaces in a row become two normal spaces. Run the result through Remove extra spaces if you want them collapsed.

Keep joiners inside emoji. A zero-width joiner glues emoji together: 👨, 👩 and 👧 with joiners between them display as one family emoji. With the option on, a joiner is kept when it sits between two emoji; a skin tone or the emoji presentation selector (U+FE0F) just before the joiner is skipped over, so 👩🏽‍💻, 🏳️‍🌈 and ❤️‍🔥 stay whole. The option also keeps the tag characters that follow 🏴 in the England, Scotland and Wales flags. With it off, every joiner between emoji goes and the family becomes three separate emoji (2 joiners removed), and the England flag loses its 6 tag characters and turns into a plain black flag 🏴.

Worked examples

Removing a character never adds a space in its place, except for the space look-alikes. If a zero-width space was the only thing between two words, the words join up, which is how they already looked on screen.

When people need it

• A CSV header that will not match. Files saved as “UTF-8 with BOM” start with U+FEFF. The Unicode FAQ notes that a BOM interferes with any format that expects specific characters at the start of a file, so a script looking for a column called id may not find it.
• Spreadsheet lookups that fail. Excel’s CLEAN function removes only control characters 0 to 31, and TRIM removes only the normal space; Microsoft’s own help pages say neither removes the no-break space (character 160). Text copied from a web page into a cell often carries one.
• Code, URLs and logins that look right but do not work. A zero-width space inside a variable name, a link or a username makes it a different string. Pasting it here shows exactly where the extra character came from.
• Text copied from chat tools, web pages, PDFs and word processors. Any of them can carry soft hyphens, no-break spaces or direction marks. The table tells you what was actually there, instead of guessing.
• Checking a character count. Hidden characters count toward limits on forms and social posts. Clean the text, then check it with the character counter.

To see exactly what changed, paste the original and the cleaned text into Compare texts.

When to keep a character

Some of these characters do a job. The Unicode Standard describes the zero-width non-joiner as a way to change how letters connect, as is sometimes required in Persian written in Arabic script. The tool keeps it when it sits between two letters of a non-Latin script, so می‌خواهم keeps its spelling. The zero-width joiner, which Indic scripts such as Hindi use to ask for a half form, is kept between two letters or marks of those scripts too, so क्‍ष comes out unchanged and nothing is reported. Direction marks keep mixed right-to-left and left-to-right text, such as an English product name inside an Arabic sentence, in the right order; removing them can reorder punctuation on screen. A soft hyphen only shows where a long word may break at the end of a line, so removing it is usually harmless. If your text is in one of these scripts, clean only the parts that need it.

Questions