CSV files and records

Test packs and checks

1. Column profile of a CSV table

   column_profile

   Summarize each column of a CSV without declared rules: missing values, distinct values, inferred type, frequent values, value shapes, lengths, quartiles, mean, standard deviation, outliers, date ranges, duplicate rows and constant or mostly empty columns.

   Runs with

   Python 3.10 or later

   Command

   python3 column_profile.py < input.json

   Code moduleMIT12 files, 62.3 KBruns commands

2. Record entry validator with normalization rules

   record_entry_validator

   Check rows of records (CSV or objects) against declared field rules for type, required, allowed values, pattern, bounds, length and unique keys, after normalizing case, spaces, digits and dates, and list each error by row.

   Runs with

   Python 3.10 or later

   Command

   python3 record_entry_validator.py < input.json

   Code moduleMIT12 files, 70.1 KBruns commands

3. Multi-source table merge with provenance and conflicts

   multi_source_table_merge

   Compile one table from several CSV sources by mapping columns to target fields, converting units, matching on key fields, taking values by source priority, recording which source gave each value and listing disagreements.

   Runs with

   Python 3.10 or later

   Command

   python3 multi_source_table_merge.py < input.json

   Code moduleMIT12 files, 65.4 KBruns commands

4. Duplicate record finder with blocking and weighted similarity

   duplicate_record_finder

   Find likely duplicate records by comparing only pairs that share a blocking key, scoring weighted field similarities (exact, normalized, token overlap, edit distance, digits, date) and grouping duplicates into clusters.

   Runs with

   Python 3.10 or later

   Command

   python3 duplicate_record_finder.py < input.json

   Code moduleMIT12 files, 62.7 KBreads files, runs commands

5. Find duplicate customer, company or contact records with blocking keys

   find_duplicate_records_with_blocking_keys

   Find duplicate customer, company or contact records (dedupe, deduplicate, deduplication, record linkage) without comparing every row with every other row: group rows by blocking keys from name, address, email and phone, compare inside groups, and report oversized groups that were skipped.

   Files

   SKILL.md

   ComponentMIT1 files, 2.7 KBnone declared

6. Score whether two records are the same entity

   score_duplicate_pairs_by_weakest_signal

   Score whether two records are the same entity (fuzzy matching): build comparison keys for name, address, email and phone, measure token overlap and sequence similarity, and let the weakest field set the confidence.

   Files

   SKILL.md

   ComponentMIT1 files, 2.5 KBnone declared

7. Turn duplicate pairs into clusters and a merge proposal with one survivor for each cluster

   propose_a_dedupe_without_deleting_rows

   Turn duplicate pairs into clusters and a merge proposal with one survivor for each cluster, keep first or keep most complete, without deleting any row.

   Files

   SKILL.md

   ComponentMIT1 files, 2.5 KBnone declared

8. Copy a table with corrections and an approved dedupe, never in place

   copy_a_table_with_corrections_never_in_place

   Apply data corrections and an approved dedupe while copying a delimited file or a SQLite table to a new target, without touching the original.

   Files

   SKILL.md

   ComponentMIT1 files, 2.7 KBreads files, writes files

9. Find malformed values in a column that should follow one format

   detect_malformed_values_by_dominant_pattern

   Find malformed values in a column that should follow one format, such as dates, postal codes or identifiers, by comparing the character pattern of each value with the dominant pattern.

   Files

   SKILL.md

   ComponentMIT1 files, 2.7 KBnone declared

10. Profile a text column before cleaning it

    profile_text_column_before_cleaning

    Profile a text column before cleaning it: count case, whitespace, Unicode, email, website, phone and legal suffix signals, find the common character patterns, and propose cleaning rules with the evidence for each.

    Files

    SKILL.md

    ComponentMIT1 files, 2.8 KBnone declared

11. Manage exception lists for data cleaning in layers with a fixed precedence

    layer_exception_catalogs_with_precedence

    Manage exception lists for data cleaning in layers with a fixed precedence: packaged general list, task folder file, casings learned from the column, rule parameters and recorded answers.

    Files

    SKILL.md

    ComponentMIT1 files, 2.7 KBreads files