Baltor Get started

Library / Code module

Equal-cohort comparison with separate coverage

equal_cohort_comparison

Compare candidates only on the rows every candidate scored, and report separately how much of the declared universe each one covered. Missing scores are never counted as zero and never dropped per candidate, so a candidate cannot look better by skipping hard rows.

Files, names only

  • ATTRIBUTION.md 3.9 KB
  • LICENSE 1.0 KB
  • README.md 2.9 KB
  • atom_cli.py 3.0 KB
  • component.json 9.2 KB
  • equal_cohort_comparison.py 4.5 KB
  • numerics.py 28.0 KB
  • schema_lite.py 8.9 KB
  • test_equal_cohort_comparison.py 3.4 KB
  • test_package.py 12.0 KB

Full files in your harness.

What it takes and gives

Runs with
Python 3.10 or later
Entry points
compare_on_common_cohort
Known answers
7 cases its tests reproduce

Cases it covers

  • missing is not zero: Counting a's two missing rows as zero mixes its failure rate into its quality score.
  • not own rows mean: Averaging b over its own four rows includes the hard rows a skipped, so the two means describe different tasks.

Limits and scope

Limits
Means only; no interval or significance test is computed (use a paired test on the cohort rows). A small cohort can describe an unrepresentative subset: always read the coverage next to the means. Row ids must match exactly across candidates; it does not reconcile differently named rows.
Guarantees
Every reported mean uses exactly the cohort rows, the same set for every candidate. Coverage is reported per candidate against the declared universe and never enters the means. With an empty cohort the means are null and the ranking is empty; nothing is imputed.
Assumes
A missing score means the candidate produced no valid score for that row; it carries no value. Rows in the cohort are comparable across candidates (same input, same scoring rule).

Details

Kind
Code module
Licence
MIT
Declared effects
runs commands
Source
github.com/alisonjieli-png/loop-engine/tools/creative_originals/evaluation_atoms/items/equal_cohort_comparison
Size
10 files, 76.8 KB
Identity
library.supply.creative_originals.f64b783bd5c48bd069805a19.6964167f851ebbec
Version
6964167f851ebbecc3b020f5f728e133f4a245ee9960e318b5628c80c3987d3d

The version is the SHA-256 of the item your harness downloads; every file in it names its own checksum.

Where it is shown

Related components

Something wrong with this item, its rights or its safety? Report it on the support page, or from your harness after a download.