AI Unicode Checks: Stop Lookalike Identifier Errors

A request names customer ACME. The extracted reference looks identical to the registered reference, but its first character is Cyrillic rather than Latin. A reviewer sees a familiar name; the receiving system sees a different string. Silently replacing that character can be just as dangerous as ignoring it.

AI Unicode checks should preserve the original, apply a field-specific comparison policy, and keep character similarity separate from identity and authorization. The model may locate a reference and explain surrounding context. It should not decide that two lookalike identifiers are interchangeable or repair one into an authorized target.

This is a recommended implementation design for business intake, not a description of a deployed customer system. Start with one controlled reference field and its authoritative register, rather than trying to sanitize every language into one alphabet.

Separate equivalent text from lookalike text

Unicode normalization and spoof detection answer different questions. Unicode Normalization Forms (UAX #15) distinguishes canonical from compatibility equivalence. NFC can give canonically equivalent representations a consistent form. NFKC also folds compatibility distinctions; the standard cautions against applying compatibility normalization indiscriminately to arbitrary text.

Neither operation means “make this string trustworthy.” A normalization policy must come from the field contract, not from a prompt to clean the input. Keep the received spelling and the comparison form separately.

Unicode Security Mechanisms (UTS #39) provides confusable-detection mechanisms. Its comparison skeletons are internal artifacts, not replacement identifiers or suitable display strings. A confusable finding is a reason to investigate; it does not establish that two records represent the same customer.

For broader identity decisions, use an evidence-based entity resolution workflow. Character checks belong before or within that comparison, not instead of it.

Write a policy for each field, not each document

A customer display name, internal account reference and destination hostname need different rules. The following matrix is a starting specification to adapt with the data owner. It is not a Unicode conformance profile or a universal security standard.

Record the policy owner, version, exact field grammar, normalization stage, comparison behavior and hold conditions. Unicode Identifiers and Syntax (UAX #31) is a useful basis for identifier design, but it does not replace the receiving application’s contract.

Do not use “remove every invisible character” as a language-neutral rule. UTS #39 describes legitimate contextual uses of joining controls. Restrictions appropriate for a machine account code may damage a person’s name or ordinary language text.

Keep the AI boundary narrow

Supply the model with the authorized source revision, field definitions and necessary context. Ask it to locate the quoted reference, identify the field it came from, and report ambiguity. Treat its output as a proposal that still needs an exact relationship to the source.

• Capture layer: retain the original artifact and decoded text, with source location and decoding errors. Do not silently replace invalid input and call it clean.
• Deterministic validator: compare source and extracted field, apply the declared policy, and report relevant code points, transformations and collisions using a tested implementation.
• AI: explain relevant context or suggest the correct review category. It cannot waive a character violation or invent the target ID.
• Reviewer: resolve missing or conflicting evidence through an approved verification channel.
• Authoritative system: select eligible records and enforce tenant, role, current-state and action permissions independently.

A structured output can still contain a visually plausible wrong reference. If extraction changes the source spelling, retain both and flag the mismatch rather than validating only the model’s repaired version.

Inspect hidden content without granting it authority

Microsoft’s ASCII-smuggling research describes a campaign that inserted invisible Unicode tag characters into lure words to evade matching. That observation supports inspecting more than the visible text; it is not proof that every model follows hidden instructions.

Any derived inspection text remains untrusted. Do not decode a suspicious sequence and feed it back as a privileged instruction. Keep the wider prompt-injection controls around retrieval, tools and authorization.

Work through a lookalike customer reference

Constructed example: the internal reference contract allows uppercase ASCII letters only. The register contains ACME, beginning with Latin capital A (U+0041). An incoming request contains a visually similar reference beginning with Cyrillic capital A (U+0410), followed by Latin C, M and E.

1. Capture: preserve the source reference and its location. The inspection view shows the first character’s code point and script beside the original text.
2. Validate: the original fails the ASCII-only reference contract. Do not replace its first character with Latin A and retry the lookup as if the request supplied a valid ID.
3. Hold: route the request to the intake owner. A possible lookalike is evidence of ambiguity, not proof of malicious intent.
4. Verify: obtain the customer reference through the organization’s established verification channel. If it confirms ACME and the requester is authorized, record the verified target separately from the invalid submitted reference.
5. Release: authorize only the specified association or next action, against the current source and target versions. A verified reference does not approve unrelated account changes.
6. Close: read back the resulting association or action receipt. Keep the original finding and correction evidence linked to it.

If confirmation never arrives, the case stays held or closes under the documented unresolved-request policy. Repeatedly asking the LLM to reinterpret the same string supplies no new identity evidence.

Persist the decision, then control recovery

Use explicit states such as captured → checked → review required → verified → release authorized → applied. Rejected, superseded and outcome-unknown states need their own owners and exit conditions. These are proposed application states, not product claims.

A compact release record should retain:

• Work item, source revision/location, original field and extracted field.
• Field-policy version, Unicode/library/data versions and any derived comparison value.
• Finding codes, relevant code-point evidence and candidate collisions.
• Verified authoritative target, reviewer or qualifying rule, evidence reference and permitted action.
• Accepted revision, action identity, destination receipt and verified final state.

Restrict access to original documents and sensitive names; an escaped diagnostic string can still contain personal data. Store only the necessary diagnostic material in ordinary logs.

Define retry eligibility and stale-result handling

An unavailable character-checking service or unsupported data version creates a technical hold. Use a bounded retry of the same check after service recovery, retaining the source and policy versions. Do not substitute model intuition for a missing validator.

A source correction, changed field policy or new authoritative target revision requires reassessment. Mark earlier results superseded and prevent late callbacks from releasing them. A reviewer rejecting a proposed correction returns the request for clarification or closes that proposal; it is not a signal to accept the uncorrected reference.

Bind dispatch to the accepted revision and exact action. Use the destination’s supported idempotency and conditional-update behavior. On a write timeout, reconcile the existing receipt and actual target state before repeating anything. If the destination cannot establish the outcome, assign reconciliation and keep release blocked; do not create another customer to escape uncertainty.

Test legitimate text as well as suspicious text

Create a labeled acceptance set before enabling downstream writes. The following tests specify desired behavior, not measured results:

• Canonically equivalent name representations follow the declared comparison policy while their originals remain available.
• The Latin/Cyrillic lookalike reference fails the machine-field grammar without being silently rewritten.
• A legitimate multilingual name is not rejected merely because the ASCII-only rule for account codes exists elsewhere.
• A compatibility transformation that creates a collision produces review rather than automatic record selection.
• Unexpected invisible content is exposed in a restricted inspection view and never acquires instruction authority.
• A model returns a cleaned identifier absent from the source: source-to-output validation catches the alteration.
• A policy change during review invalidates pending release; a late result cannot apply the old decision.
• A timeout after successful dispatch reconciles to one applied action.

Measure wrong-record releases, silent identifier alterations, false holds on legitimate text, unresolved-case age and time to verified closure. Report by field, language/script and source type, using appropriately protected data. A high overall pass rate can hide systematic rejection of one language community. Set release criteria from consequences and representative tests, not a generic accuracy percentage.

Where Workhint fits

The human exception needs clear ownership. Workhint’s Workflow Engine describes routing approvals to defined roles and requiring details, files, notes or approval steps before work advances. Those capabilities are relevant to collecting reference-verification evidence and assigning its reviewer.

The workflow automation software solution describes structured forms, conditional routing and review stages with assigned approvers. Use that process-design scope when discussing how an identifier exception reaches the right owner.

The Unicode validator, confusable detector, authoritative lookup, version fence and external write guarantees in this article require separate implementation and capability checks. These pages do not establish a native Unicode security scanner or prove that an identifier is authentic.

Frequently asked questions

Does NFC normalization stop identifier spoofing?

No. Canonical normalization addresses equivalent representations, not trust or every visual similarity. Validate the field contract, detect relevant ambiguity and independently verify the target and requester authority.

Should we replace confusable characters with their skeleton?

No. Keep skeletons inside the confusability-checking process. They should not become stored customer IDs, replacement names or a shortcut to automatic association.

Should an AI workflow reject every mixed-script name?

No. A machine reference may have a restrictive grammar, but a display name serves a different purpose. Apply a documented field policy and test legitimate language use rather than treating non-Latin text as evidence of an attack.

What is a practical first deployment?

Choose one internal reference with an explicit contract and a named data owner. Run the checks without live writes, test both legitimate and ambiguous inputs, and enable only the bounded action whose review and reconciliation paths are proven.