# CSV files and records

Part of Corner cases by Baltor.

Column profiles, field rule checks, merges of several sources and duplicate finders for CSV records. Corrections go to a new copy, never into the original file.

## What to watch for

- An empty cell, an empty string and the text NULL are three different values.
- Duplicates rarely match exactly, so compare normalized keys within blocks.
- Correct a copy of the table and keep the original for comparison.

## Components

- [Column profile of a CSV table](https://baltor.ai/components/column-profile-of-a-csv-table-a07efb6d): Summarize each column of a CSV without declared rules: missing values, distinct values, inferred type, frequent values, value shapes, lengths, quartiles, mean, standard deviation, outliers, date ranges, duplicate rows and constant or mostly empty columns. Name `column_profile`, licence MIT, 12 files.
- [Record entry validator with normalization rules](https://baltor.ai/components/record-entry-validator-with-normalization-rules-c7330118): Check rows of records (CSV or objects) against declared field rules for type, required, allowed values, pattern, bounds, length and unique keys, after normalizing case, spaces, digits and dates, and list each error by row. Name `record_entry_validator`, licence MIT, 12 files.
- [Multi-source table merge with provenance and conflicts](https://baltor.ai/components/multi-source-table-merge-with-provenance-and-conflicts-8ad96aa9): Compile one table from several CSV sources by mapping columns to target fields, converting units, matching on key fields, taking values by source priority, recording which source gave each value and listing disagreements. Name `multi_source_table_merge`, licence MIT, 12 files.
- [Duplicate record finder with blocking and weighted similarity](https://baltor.ai/components/duplicate-record-finder-with-blocking-and-weighted-simil-6431b478): Find likely duplicate records by comparing only pairs that share a blocking key, scoring weighted field similarities (exact, normalized, token overlap, edit distance, digits, date) and grouping duplicates into clusters. Name `duplicate_record_finder`, licence MIT, 12 files.
- [Find duplicate customer, company or contact records with blocking keys](https://baltor.ai/components/find-duplicate-records-with-blocking-keys): Find duplicate customer, company or contact records (dedupe, deduplicate, deduplication, record linkage) without comparing every row with every other row: group rows by blocking keys from name, address, email and phone, compare inside groups, and report oversized groups that were skipped. Name `find_duplicate_records_with_blocking_keys`, licence MIT, 1 files.
- [Score whether two records are the same entity](https://baltor.ai/components/score-duplicate-pairs-by-weakest-signal): Score whether two records are the same entity (fuzzy matching): build comparison keys for name, address, email and phone, measure token overlap and sequence similarity, and let the weakest field set the confidence. Name `score_duplicate_pairs_by_weakest_signal`, licence MIT, 1 files.
- [Turn duplicate pairs into clusters and a merge proposal with one survivor for each cluster](https://baltor.ai/components/propose-a-dedupe-without-deleting-rows): Turn duplicate pairs into clusters and a merge proposal with one survivor for each cluster, keep first or keep most complete, without deleting any row. Name `propose_a_dedupe_without_deleting_rows`, licence MIT, 1 files.
- [Copy a table with corrections and an approved dedupe, never in place](https://baltor.ai/components/copy-a-table-with-corrections-never-in-place): Apply data corrections and an approved dedupe while copying a delimited file or a SQLite table to a new target, without touching the original. Name `copy_a_table_with_corrections_never_in_place`, licence MIT, 1 files.
- [Find malformed values in a column that should follow one format](https://baltor.ai/components/detect-malformed-values-by-dominant-pattern): Find malformed values in a column that should follow one format, such as dates, postal codes or identifiers, by comparing the character pattern of each value with the dominant pattern. Name `detect_malformed_values_by_dominant_pattern`, licence MIT, 1 files.
- [Profile a text column before cleaning it](https://baltor.ai/components/profile-text-column-before-cleaning): Profile a text column before cleaning it: count case, whitespace, Unicode, email, website, phone and legal suffix signals, find the common character patterns, and propose cleaning rules with the evidence for each. Name `profile_text_column_before_cleaning`, licence MIT, 1 files.
- [Manage exception lists for data cleaning in layers with a fixed precedence](https://baltor.ai/components/layer-exception-catalogs-with-precedence): Manage exception lists for data cleaning in layers with a fixed precedence: packaged general list, task folder file, casings learned from the column, rule parameters and recorded answers. Name `layer_exception_catalogs_with_precedence`, licence MIT, 1 files.

Your agent downloads these through Baltor with Pro, $49 a month: https://baltor.ai/get-started