Baltor Get started

Library / Code module

Duplicate record finder with blocking and weighted similarity

duplicate_record_finder

Find likely duplicate records by comparing only pairs that share a blocking key, scoring weighted field similarities (exact, normalized, token overlap, edit distance, digits, date) and grouping duplicates into clusters.

Files, names only

  • ATTRIBUTION.md 4.9 KB
  • LICENSE 1.0 KB
  • ONET-ATTRIBUTION.md 4.8 KB
  • PROCEDURE.md 2.6 KB
  • README.md 5.0 KB
  • TOOLS.md 2.0 KB
  • component.json 11.0 KB
  • duplicate_record_finder.py 10.6 KB
  • examples/known_good.json 2.3 KB
  • examples/known_wrong.json 436 bytes
  • kit_schema.py 11.4 KB
  • test_package.py 6.8 KB

Full files in your harness.

What it takes and gives

Runs with
Python 3.10 or later
Command
python3 duplicate_record_finder.py < input.json
Function
duplicate_record_finder.run
Required input
records, fields, blocking

Cases it covers

  • Known wrong example: Two records share the id V1, so a duplicate decision could not say which record it means. Refused as duplicate id.
  • input invalid: refused with exit code 2
  • input not json: refused with exit code 2
  • duplicate id: refused with exit code 2
  • field absent: refused with exit code 2
  • weights invalid: refused with exit code 2
  • thresholds out of order: refused with exit code 2
  • too many pairs: refused with exit code 2

Limits and scope

Limits
Pairs that share no blocking key are never compared, so weak blocking misses duplicates. Edit distance uses the first 200 characters. Similarity weights and thresholds are judgment; tune them on a labelled sample. Clusters chain pairs, so A like B and B like C puts A and C together even when A and C differ. The helper never merges records; it only proposes.
Work activities
Verify accuracy of records; Maintain records, documents, or other files; Maintain records of customer accounts.

Details

Kind
Code module
Licence
MIT
Declared effects
reads files, runs commands
Source
github.com/alisonjieli-png/loop-engine/tools/creative_originals/occupation_task_kits/items/duplicate_record_finder
Size
12 files, 62.7 KB
Identity
library.supply.creative_originals.155023e1c0dc7d674b9d7dd2.6431b4781647eb11
Version
6431b4781647eb11733322182dabd154e7c9a121a65758ebbfa624e5eea3faf2

The version is the SHA-256 of the item your harness downloads; every file in it names its own checksum.

Where it is shown

Related components

Something wrong with this item, its rights or its safety? Report it on the support page, or from your harness after a download.