Multicalibration audit by prediction bin and group

What it takes and gives

Runs with

Python 3.10 or later

Entry points

multicalibration_audit, multicalibrate

Known answers

6 cases its tests reproduce

Limits and scope

Limits

Equal-width bins only; residuals in sparse cells are noisy and no confidence interval is computed. The audit covers only the declared groups, not every computable subgroup. Patching changes predictions on the audit data itself: evaluate the patched predictor on fresh data before trusting it.

Guarantees

Each cell's residual is the exact mean of outcome minus prediction over its rows; the implicit group all is always audited. Each patch of a flagged cell lowers the squared error by at least minimum_count x tolerance^2, so multicalibrate stops; on return without hitting max_rounds no cell is flagged. A prediction of exactly 1 joins the last bin.

Assumes

Groups are declared in advance; the audit only checks the groups it is given. Cells smaller than minimum_count are reported but never flagged; their residuals are noisy.

Where it is shown

• Statistics for comparing resultsDeterministic skills

Related components

• Brier score decomposition and expected calibration error
• Split conformal prediction intervals
• Anytime-valid confidence sequence for a bounded mean
• Probabilistic and deflated Sharpe ratio
• Ledoit-Wolf covariance shrinkage
• Maximum mean discrepancy with an RBF kernel