Baltor Get started

Library / Code module

Mechanism-held-out splits for synthetic task suites

mechanism_heldout_split

Split a synthetic task suite by generating mechanism so that no template, rule family or simulator appears on both sides, build balanced mechanism-level folds, and detect leakage in any given split. Measures transfer to unseen mechanisms instead of recall of seen ones.

Files, names only

  • ATTRIBUTION.md 3.9 KB
  • LICENSE 1.0 KB
  • README.md 2.8 KB
  • atom_cli.py 3.0 KB
  • component.json 10.5 KB
  • mechanism_heldout_split.py 5.6 KB
  • numerics.py 28.0 KB
  • schema_lite.py 8.9 KB
  • test_mechanism_heldout_split.py 3.4 KB
  • test_package.py 12.0 KB

Full files in your harness.

What it takes and gives

Runs with
Python 3.10 or later
Entry points
split_by_mechanism, group_k_folds, leakage
Known answers
5 cases its tests reproduce

Cases it covers

  • task level split is not clean: Splitting tasks at random puts generators m1 and m3 on both sides, so the test side measures recall of seen mechanisms.

Limits and scope

Limits
Mechanism labels are trusted as given; mechanisms that share hidden structure (two templates with the same solution procedure) still leak. Greedy fold assignment balances task counts approximately, not optimally. Holding out whole mechanisms leaves fewer, larger test units, so estimates have more variance than task-level splits.
Guarantees
split_by_mechanism holds out k = max(1, min(M - 1, floor(fraction M + 0.5))) whole mechanisms chosen by a seeded Fisher-Yates shuffle of the sorted names; no mechanism is on both sides. group_k_folds partitions every task and every mechanism into exactly one fold, assigning mechanisms largest first to the least loaded fold. leakage lists every mechanism present on both sides of a split.
Assumes
Each task is labelled with the mechanism that generated it, and tasks of different mechanisms do not share hidden structure. Seeded shuffles use random.Random.random() only, so a seed gives the same split on every Python version.

Details

Kind
Code module
Licence
MIT
Declared effects
runs commands
Source
github.com/alisonjieli-png/loop-engine/tools/creative_originals/evaluation_atoms/items/mechanism_heldout_split
Size
10 files, 79.1 KB
Identity
library.supply.creative_originals.44c5bd74ff487e1b822ae423.437c9550773b47a2
Version
437c9550773b47a2bd9da10314dc1c0220e19c532f667a108764b26201d668a6

The version is the SHA-256 of the item your harness downloads; every file in it names its own checksum.

Where it is shown

Related components

Something wrong with this item, its rights or its safety? Report it on the support page, or from your harness after a download.