# Text, Unicode and encodings

Part of Corner cases by Baltor.

Look-alike Unicode characters, Unicode blocks and emoji tables, base64 and base58 validators, and normalizers for whitespace, Unicode and personal data in free text.

## What to watch for

- Cyrillic а and Latin a look the same and compare as different strings.
- A non-breaking space looks like a space and survives a plain split.
- One emoji can be several code points joined, so its length is not one.

## Components

- [Unicode homoglyphs mapped to their ASCII-like base characters](https://baltor.ai/components/unicode-homoglyphs-mapped-to-their-ascii-like-base-chara-0a7acf78): Unicode homoglyphs mapped to their base (ASCII-like) characters for confusable detection: 6311 rows with a JSON Schema, a checked loader and tests. Name `data.json`, licence MIT, 8 files.
- [Unicode blocks with start and end code points and the Unicode version that introduced them](https://baltor.ai/components/unicode-blocks-with-start-and-end-code-points-and-the-un-b49d2640): Unicode blocks with start and end code points and the Unicode version that introduced them: 346 rows with a JSON Schema, a checked loader and tests. Name `unicode-blocks.json`, licence MIT, 8 files.
- [Unicode character names of every emoji by hexadecimal code point sequence](https://baltor.ai/components/unicode-character-names-of-every-emoji-by-hexadecimal-co-5d2b388f): Unicode character names of every emoji by hexadecimal code point sequence: 3979 rows with a JSON Schema, a checked loader and tests. Name `unicode-names.json`, licence MIT, 8 files.
- [Emoji hexcodes introduced in each Unicode version](https://baltor.ai/components/emoji-hexcodes-introduced-in-each-unicode-version-25123c0c): Emoji hexcodes introduced in each Unicode version: 15 rows with a JSON Schema, a checked loader and tests. Name `unicode.json`, licence MIT, 8 files.
- [base64 from validators input validation utilities](https://baltor.ai/components/base64-from-validators-input-validation-utilities-7a1588e7): base64 from validators input validation utilities: Return whether or not given value is a valid base64 encoding. Name `base64`, licence MIT, 6 files.
- [base58 from validators input validation utilities](https://baltor.ai/components/base58-from-validators-input-validation-utilities-b64861ab): base58 from validators input validation utilities: Return whether or not given value is a valid base58 encoding. Name `base58`, licence MIT, 6 files.
- [ensure_unicode from xdoctest](https://baltor.ai/components/ensure-unicode-from-xdoctest-2b8d9c0d): ensure_unicode from xdoctest (docstring parsing and text utilities): Casts bytes into utf8 (mostly for python2 compatibility) One tested Python function, copied with the code it needs. Name `ensure_unicode`, licence MIT AND Apache-2.0, 6 files.
- [decode_url from persian](https://baltor.ai/components/decode-url-from-persian-29ad30ec): decode_url from persian (Persian text normalisation): Decode percent-encoded Persian characters in URLs. Name `decode_url`, licence MIT, 6 files.
- [Normalize whitespace and Unicode in text fields](https://baltor.ai/components/normalize-whitespace-and-unicode-text): Normalize whitespace and Unicode in text fields: non-breaking spaces, tabs, doubled spaces, split accents, zero-width characters and typographic quotes. Name `normalize_whitespace_and_unicode_text`, licence MIT, 1 files.
- [Group byte-identical UTF-8 text records once while keeping every source reference](https://baltor.ai/components/baltor-exact-content-groups): Group byte-identical UTF-8 text records once while keeping every source reference and its provenance, reversibly and without merging identities or deleting anything. Name `baltor_exact_content_groups`, licence MIT, 9 files.
- [Personal data masker for free text](https://baltor.ai/components/personal-data-masker-for-free-text-2636cba2): Replace personal identifiers in free text (emails, web addresses, IP addresses, card numbers, phone numbers, dates, ID numbers, ZIP codes, listed names and custom patterns) with numbered placeholders and return an inventory without the values. Name `personal_data_masker`, licence MIT, 12 files.

Your agent downloads these through Baltor with Pro, $49 a month: https://baltor.ai/get-started