Text, Unicode and encodings

Test packs and checks

1. Unicode homoglyphs mapped to their ASCII-like base characters

   data.json

   Unicode homoglyphs mapped to their base (ASCII-like) characters for confusable detection: 6311 rows with a JSON Schema, a checked loader and tests.

   Rows

   6311

   Data file

   data/data.json

   Code moduleMIT8 files, 110.2 KBreads files, runs commands

2. Unicode blocks with start and end code points and the Unicode version that introduced them

   unicode-blocks.json

   Unicode blocks with start and end code points and the Unicode version that introduced them: 346 rows with a JSON Schema, a checked loader and tests.

   Rows

   346

   Data file

   data/unicode-blocks.json

   Code moduleMIT8 files, 36.9 KBreads files, runs commands

3. Unicode character names of every emoji by hexadecimal code point sequence

   unicode-names.json

   Unicode character names of every emoji by hexadecimal code point sequence: 3979 rows with a JSON Schema, a checked loader and tests.

   Rows

   3979

   Data file

   data/unicode-names.json

   Code moduleMIT8 files, 258.1 KBreads files, runs commands

4. Emoji hexcodes introduced in each Unicode version

   unicode.json

   Emoji hexcodes introduced in each Unicode version: 15 rows with a JSON Schema, a checked loader and tests.

   Rows

   15

   Data file

   data/unicode.json

   Code moduleMIT8 files, 137.7 KBreads files, runs commands

5. base64 from validators input validation utilities

   base64

   base64 from validators input validation utilities: Return whether or not given value is a valid base64 encoding.

   Function

   base64

   Project

   validators input validation utilities

   Code moduleMIT6 files, 9.3 KBruns commands

6. base58 from validators input validation utilities

   base58

   base58 from validators input validation utilities: Return whether or not given value is a valid base58 encoding.

   Function

   base58

   Project

   validators input validation utilities

   Code moduleMIT6 files, 9.2 KBruns commands

7. ensure_unicode from xdoctest

   ensure_unicode

   ensure_unicode from xdoctest (docstring parsing and text utilities): Casts bytes into utf8 (mostly for python2 compatibility) One tested Python function, copied with the code it needs.

   Function

   ensure_unicode

   Project

   xdoctest (docstring parsing and text utilities)

   Code moduleMIT AND Apache-2.06 files, 15.9 KBruns commands

8. decode_url from persian

   decode_url

   decode_url from persian (Persian text normalisation): Decode percent-encoded Persian characters in URLs.

   Function

   decode_url

   Project

   persian (Persian text normalisation)

   Code moduleMIT6 files, 5.9 KBruns commands

9. Normalize whitespace and Unicode in text fields

   normalize_whitespace_and_unicode_text

   Normalize whitespace and Unicode in text fields: non-breaking spaces, tabs, doubled spaces, split accents, zero-width characters and typographic quotes.

   Files

   SKILL.md

   ComponentMIT1 files, 2.6 KBnone declared

10. Group byte-identical UTF-8 text records once while keeping every source reference

    baltor_exact_content_groups

    Group byte-identical UTF-8 text records once while keeping every source reference and its provenance, reversibly and without merging identities or deleting anything.

    Input fields

    version, component, input

    Output fields

    version, component, output

    SkillMIT9 files, 55.8 KBreads files, runs commands

11. Personal data masker for free text

    personal_data_masker

    Replace personal identifiers in free text (emails, web addresses, IP addresses, card numbers, phone numbers, dates, ID numbers, ZIP codes, listed names and custom patterns) with numbered placeholders and return an inventory without the values.

    Runs with

    Python 3.10 or later

    Command

    python3 personal_data_masker.py < input.json

    Code moduleMIT12 files, 70.3 KBruns commands