Invisible zero-width characters that break code (U+200B, BOM, bidi)
The characters are there. Your eyes just cannot see them — until a password fails, a test flakes, or a phishing lookalike slips past a review.
The usual suspects
| Character | Code point | UTF-8 bytes | Why it hurts |
|---|---|---|---|
| Zero-width space (ZWSP) | U+200B | E2 80 8B | Breaks identifiers, URLs, "equal" strings that look identical |
| Zero-width non-joiner / joiner | U+200C / U+200D | E2 80 8C / 8D | ZWNJ surprises in names; ZWJ builds emoji sequences |
| BOM / zero-width no-break space | U+FEFF | EF BB BF (UTF-8 BOM) | File starts with invisible bytes; JSON parsers and shebangs choke |
| Soft hyphen | U+00AD | C2 AD | Copied from Word/PDF; appears only at line wraps |
| Non-breaking space | U+00A0 | C2 A0 | Looks like a normal space; fails == " " and some split/trim |
| Bidi overrides (RLO/LRO/PDF…) | U+202A–U+202E, U+2066–U+2069 | varies | Can visually reverse text — Trojan Source / phishing lookalikes |
| Tag characters | U+E0001, U+E0020–U+E007F | 4-byte UTF-8 | Mostly obsolete tags; can hide in "empty" looking strings |
Interactive emoji / ZWJ walkthrough: Wirebook Who stole the emojis?. Invisible-character lens: Wirebook ghost footprints. Machine-readable report for any paste: Anvil /v1/text/inspect (flags invisible, bom, bidi_override, homoglyph).
How they sneak in
- Copy-paste from Slack, Teams, email, Notion, Word, Google Docs, PDFs — soft hyphens, NBSP and ZWSP ride along with "clean" looking text.
- Web scrapes and OCR — layout engines insert soft hyphens and bidi marks.
- Phishing / lookalike domains and code reviews — Latin
avs Cyrillicа, or bidi overrides that makeimportappear reversed (Trojan Source). Homoglyphs are different from zero-width, but the same X-ray catches both. - Editors "helpful" UTF-8 BOM on Windows when saving scripts that must start with
#or{.
How to spot them
- Editor: VS Code "Render Whitespace" / "Unicode Highlight"; Vim
:set list; hex mode. - Hex / bytes:
xxd file | head— a UTF-8 BOM isef bb bfat byte 0. Paste into Base64 (UTF-8) to see the bytes as text. - Lengths disagree: word & character counter vs what you count by eye. JavaScript
str.lengthcounts UTF-16 units; Pythonlen(s)counts code points — both will be "too high" when invisibles are present. - Diff: paste "same" strings into the text diff — invisibles show as changes even when glyphs match.
- API:
POST https://api.oddlathe.com/v1/text/inspectwith your string — see docs. - Regex (strip common offenders):
# Python
import re
cleaned = re.sub(r"[\u200B-\u200D\uFEFF\u00AD\u2060-\u2064\u2066-\u2069\u202A-\u202E]", "", text)
cleaned = cleaned.replace("\u00a0", " ") # NBSP → normal space
# JavaScript
const cleaned = text
.replace(/[\u200B-\u200D\uFEFF\u00AD\u2060-\u2064\u2066-\u2069\u202A-\u202E]/g, "")
.replace(/\u00a0/g, " ");Do not blindly strip U+200D (ZWJ) from user-facing emoji text — that is what glues family/profession emoji together (see the tofu guide). Strip ZWJ only in identifiers, tokens and security-sensitive comparisons.
Fix it at the source
- Save source files as UTF-8 without BOM.
- Normalise user input (NFC + strip zero-widths) before auth checks, cache keys and unique indexes.
- In CI, fail on unexpected bidi overrides in source (Trojan Source scanners) and on UTF-8 BOM in
.js/.json/.py. - If the text looks like gibberish rather than invisible, you may have mojibake instead.
- Mystery paste? What is this?
Related
Why one hidden character halves a text message · SMS length counter · Removing spaces and odd characters from file names · Invisible characters in a password · Strings that look the same but are not equal · Non-breaking space vs space · Look-alike (homoglyph) characters · Mojibake / ðŸ / é · Emoji shows as □ / tofu · MySQL Incorrect string value / utf8mb4 · Wirebook emoji lesson · Anvil text X-ray API
Questions
What is a zero-width space?
U+200B is a character that takes no space on screen but still counts in string length and breaks identifiers, URLs and equality checks. It often arrives via copy-paste.
Why does my JSON file fail to parse at the first character?
A UTF-8 BOM (U+FEFF, bytes EF BB BF) at the start is a common cause. Re-save as UTF-8 without BOM.
How do I remove invisible Unicode in Python or JavaScript?
Use a regex over U+200B–U+200D, U+FEFF, soft hyphen, and bidi marks; map U+00A0 to a normal space. Keep ZWJ if you need intact emoji sequences.
Are bidi overrides dangerous?
They can visually reorder source code or URLs (Trojan Source / phishing). Flag U+202A–U+202E and U+2066–U+2069 in reviews and CI.
Is a non-breaking space the same as a space?
No. U+00A0 looks identical but is a different code point, so trim/split/equality can fail. Normalise it to U+0020 when comparing.
How can I inspect a string for invisibles?
Use Anvil text X-ray (https://api.oddlathe.com/docs/text-inspect), an editor Unicode highlight, xxd/hex dump, or the word-counter and text-diff tools.
Published · updated .