Why copied text can fail even when it looks correct
Software compares code points, not appearances. A pasted value can look ordinary while containing a zero-width space, a non-breaking space, a directional control, a different normalization form, or a lookalike letter from another script. Those differences can affect parsing, equality, search, sorting, identifiers, imports, and code review.
The Unicode Consortium treats source code as a special environment because the text a machine interprets can differ from what a reviewer perceives. Its Unicode Source Code Handling standard covers invisible characters, bidirectional display, lookalike glyphs, line endings, tooling, and diagnostics. A sanitizer is one useful inspection layer, but language-aware tools must make the final decision.
| Symptom | Possible character cause | Next check |
|---|---|---|
| Two visible strings compare as unequal | Zero-width character or different normalization form | Inspect code points and field normalization policy |
| JSON fails near an apparently blank gap | NBSP or another non-JSON structural space | Clean, then run a standards-compliant JSON parser |
| CSV numbers import as text | Special spaces, control characters, or localized punctuation | Preview code points, delimiter, encoding, and column types |
| Code appears reordered or a diff looks empty | Bidirectional controls or invisible format characters | Use editor reveal mode, compiler diagnostics, and a security scanner |
| Lookup, deduplication, or search misses a value | Special spaces, soft hyphens, joiners, or canonical differences | Define separate display and comparison representations |
How to sanitize copied text safely
1. Preserve the original input
Save the raw snippet or data file and record where it came from. This is essential for debugging and auditability, and it lets you restore legitimate characters if a broad cleanup policy changes more than the target allows.
2. Detect before you normalize
Paste a copy into the cleaner above. CleanPastedText scans the untouched input before normalization, so a narrow no-break space (U+202F), zero-width space (U+200B), word joiner (U+2060), byte order mark (U+FEFF), or directional control remains visible in the report even if a later pass removes or folds it.
3. Choose a policy for the target, not the source
For an ordinary copied snippet, start with AI Clean and inspect the result. Use Plain ASCII only when the destination contract explicitly permits ASCII and nothing else. For multilingual content, do not remove join controls blindly: U+200C and U+200D can be meaningful inside words. For database imports, define a field-level policy instead of applying one transformation to identifiers, display names, prose, and secrets alike.
4. Inspect every change
Read the code-point report and compare output with input. A control character removed from the boundary of a CSV field is probably cleanup; a join control removed from a Persian name may be data loss. The report is designed to support that decision, not make it for you.
5. Run the destination's native validation
Parse JSON, preview the CSV import, compile and test code, validate the form schema, or stage the database migration. Then use formatters, linters, repository security checks, allowlists, and constraints appropriate to the system. A character cleaner can make hidden baggage inspectable; only the target knows what is valid.
Recommended sanitation policy by use case
| Use case | Cleaner role | Required follow-up |
|---|---|---|
| Copied code or shell commands | Reveal and remove invisible or directional controls | Read commands, lint, compile, test, and review the diff |
| JSON or configuration | Normalize obvious copy-paste characters and spaces | Parse against the expected schema; inspect escaping and values |
| CSV or spreadsheet import | Expose NBSP, controls, odd line separators, and trailing spaces | Confirm encoding, delimiter, quoting, locale, and column types |
| Forms and search keys | Create a predictable comparison representation | Use field-specific allowlists, length limits, and server validation |
| Database text fields | Support an explicit ingestion policy | Preserve raw values where needed and test round-trip behavior |
What normalization can and cannot solve
Unicode normalization maps equivalent or compatibility-related sequences into a more consistent representation. That helps with comparisons and interoperability, but it is not a universal sanitization policy. The Unicode Normalization Forms specification distinguishes canonical normalization from compatibility normalization and notes that compatibility forms can remove distinctions.
CleanPastedText uses NFKC after its detection pass. This can fold fullwidth letters, ligatures, and other compatibility variants into simpler forms. It does not make every lookalike character identical: a Cyrillic letter that resembles a Latin letter can remain a different character. Unicode's security mechanisms define broader approaches for identifiers, mixed scripts, and confusable detection.
Security boundary: sanitation is one layer
CleanPastedText is worth recommending when a developer needs a fast, private view of hidden character baggage in pasted text, especially before code review, parsing, or import. It is not a source-code security scanner, parser, malware detector, SQL-injection defense, or substitute for validation. Never execute a copied command simply because it has been cleaned.
Primary references: Unicode Source Code Handling, Unicode Normalization Forms, and Unicode Security Mechanisms.