The Regex That Actually Removes Invisible Unicode Characters

What regex actually removes invisible Unicode characters?

A property escape over Unicode's Format category, not a whitespace shorthand. Zero-width spaces, word joiners, byte order marks, soft hyphens, and bidirectional controls all share one thing: the Unicode Character Database classifies every one of them as general category Cf ("Format"). A single regex can target the whole group at once:

function stripInvisible(text) {
  return text
    // Cf: zero-width space/joiner, BOM, soft hyphen, bidi controls, tag
    // characters — except ZWNJ/ZWJ, which change spelling in some scripts.
    .replace(/\p{Cf}/gu, (ch) => (ch === "\u200C" || ch === "\u200D" ? ch : ""))
    // Zs: fold exotic Unicode spaces to a plain one instead of deleting them.
    .replace(/\p{Zs}/gu, (ch) => (ch === " " ? ch : " "));
}

The u flag is not optional, and forgetting it fails silently rather than loudly. Unicode property escapes were added in ES2018 and only exist in Unicode-aware mode; drop the flag and \p{Cf} is parsed under the older Annex B grammar as the identity escape \p (a literal "p") followed by the literal text {Cf}. No error, no warning — the pattern just matches the four-character string p{Cf} and quietly ignores every real invisible character in the input.

Why doesn't \s already catch this?

Because whitespace and "format character" are different Unicode concepts, and \s only ever matches the first one. JavaScript's \s is defined against a specific WhiteSpace list that happens to include the non-breaking space (U+00A0) and, for historical script-tag reasons, the byte order mark (U+FEFF) — which is why those two feel like they get cleaned up "for free." A zero-width space (U+200B) or word joiner (U+2060) is category Cf, not whitespace by any definition, so \s walks straight past it no matter how many times you apply it. Python's \s behaves the same way for the same reason: it tracks Unicode whitespace, not the Format category.

Removing invisible characters in Python

Python's built-in re module has no \p{...} syntax at all — porting a JavaScript or PCRE pattern verbatim raises re.error: bad escape \p at compile time, since re has no concept of a Unicode property test. Two ways around it:

Stdlib only, filtering by category name directly:

import unicodedata

def strip_invisible(text: str) -> str:
    keep_anyway = "\u200c\u200d\t\n\r"  # ZWNJ, ZWJ, and real whitespace
    return "".join(
        ch for ch in text
        if unicodedata.category(ch) not in ("Cf", "Cc") or ch in keep_anyway
    )

unicodedata.category(ch) returns the exact two-letter general category for a single character — the same value the Unicode Character Database itself assigns — so this needs no external dependency and stays accurate as long as the Python interpreter's bundled Unicode version does.

With the third-party regex package (not the standard library — pip install regex), the same \p{Cf} syntax as JavaScript works directly:

import regex

def strip_invisible(text: str) -> str:
    return regex.sub(r"(?V1)[\p{Cf}--[\u200c\u200d]]", "", text)

-- inside the character class is set subtraction — one of the regex package's own extensions over re, gated behind the (?V1) flag shown at the start of the pattern (the module defaults to VERSION0, which only understands plain sets). Plain re and native JavaScript regex have no set-subtraction equivalent at all, which is why the JavaScript version above filters ZWNJ/ZWJ back in with a callback instead.

What a Cf/Zs regex still misses

Unicode's general categories are exact, but there are more of them than "format" and "space." Two gaps matter most in practice:

Neither omission is a bug in the regex — it's doing exactly what a Format-category test is supposed to do. It just means "strip Cf" and "strip every invisible or hidden-data character" are two different jobs, and conflating them is how a homemade cleaner can report success on text that still carries a payload.