Baltor Get started

Library / Code module

Brier score decomposition and expected calibration error

brier_decomposition_calibration_error

Split the Brier score of probability forecasts into reliability, resolution, uncertainty and the two within-bin terms that make the decomposition exact for binned forecasts, and compute the expected calibration error. Use it to tell miscalibration apart from lack of discrimination.

Files, names only

  • ATTRIBUTION.md 4.0 KB
  • LICENSE 1.0 KB
  • README.md 2.5 KB
  • atom_cli.py 3.0 KB
  • brier_decomposition_calibration_error.py 4.7 KB
  • component.json 9.6 KB
  • numerics.py 28.0 KB
  • schema_lite.py 8.9 KB
  • test_brier_decomposition_calibration_error.py 2.3 KB
  • test_package.py 12.0 KB

Full files in your harness.

What it takes and gives

Runs with
Python 3.10 or later
Entry points
brier_decomposition, expected_calibration_error
Known answers
10 cases its tests reproduce

Cases it covers

  • three terms are not the score: REL - RES + UNC equals the Brier score only when forecasts inside a bin are identical; here it overstates it by 0.0133.

Limits and scope

Limits
Equal-width bins only; reliability, resolution and ECE all depend on the binning, and ECE is biased upward for small bins. Binary outcomes only. No confidence intervals are computed.
Guarantees
brier = reliability - resolution + uncertainty + within_bin_variance - within_bin_covariance up to rounding; identity_residual reports the difference. With identical forecasts inside each bin the two within-bin terms are 0 and the three-term Murphy decomposition is recovered. ECE is the count-weighted mean absolute gap between bin mean forecast and bin outcome rate.
Assumes
Outcomes are binary and forecasts are probabilities of outcome 1. Bins are equal width on [0, 1]; a forecast of exactly 1 joins the last bin.

Details

Kind
Code module
Licence
MIT
Declared effects
runs commands
Source
github.com/alisonjieli-png/loop-engine/tools/creative_originals/evaluation_atoms/items/brier_decomposition_calibration_error
Size
10 files, 75.9 KB
Identity
library.supply.creative_originals.12ec014894c83d8bbd49b4c2.9e958209c99b3896
Version
9e958209c99b3896be945659c89f11cff01ea879329179e3647ba63a16582625

The version is the SHA-256 of the item your harness downloads; every file in it names its own checksum.

Where it is shown

Related components

Something wrong with this item, its rights or its safety? Report it on the support page, or from your harness after a download.