Statistics for comparing results

Components

1. Brier score decomposition and expected calibration error

   brier_decomposition_calibration_error

   Split the Brier score of probability forecasts into reliability, resolution, uncertainty and the two within-bin terms that make the decomposition exact for binned forecasts, and compute the expected calibration error.

   Runs with

   Python 3.10 or later

   Entry points

   brier_decomposition, expected_calibration_error

   Code moduleMIT10 files, 75.9 KBruns commands

2. Split conformal prediction intervals

   split_conformal_intervals

   Turn any point predictor into prediction intervals with a finite-sample coverage guarantee, using held-out calibration residuals and the exact ceil((n + 1)(1 - alpha)) quantile index, optionally per group (Mondrian), with a seeded coverage check on held-out data.

   Runs with

   Python 3.10 or later

   Entry points

   conformal_quantile, split_conformal, coverage, coverage_simulation

   Code moduleMIT10 files, 79.5 KBruns commands

3. Anytime-valid confidence sequence for a bounded mean

   anytime_confidence_sequence

   Give confidence intervals for the mean of bounded observations that hold at every time simultaneously, so results can be monitored after each run and stopped at any point.

   Runs with

   Python 3.10 or later

   Entry points

   hoeffding_sequence, betting_sequence, pointwise_wilson_sequence, coverage_simulation

   Code moduleMIT10 files, 87.0 KBruns commands

4. Probabilistic and deflated Sharpe ratio

   deflated_sharpe_ratio

   Compute the probability that a strategy's true Sharpe ratio exceeds a benchmark given skewness, kurtosis and sample length, and deflate it for having picked the best of many trials by comparing against the expected maximum Sharpe ratio of pure-luck trials.

   Runs with

   Python 3.10 or later

   Entry points

   sharpe_moments, probabilistic_sharpe_ratio, expected_maximum_sharpe, expected_maximum_of_normals, deflated_sharpe_ratio

   Code moduleMIT10 files, 77.3 KBruns commands

5. Ledoit-Wolf covariance shrinkage

   ledoit_wolf_shrinkage

   Shrink a sample covariance toward a scaled identity with the Ledoit-Wolf intensity, giving a well-conditioned, positive definite estimate even when there are fewer observations than variables.

   Runs with

   Python 3.10 or later

   Entry points

   ledoit_wolf, shrink

   Code moduleMIT10 files, 75.8 KBruns commands

6. Maximum mean discrepancy with an RBF kernel

   rbf_maximum_mean_discrepancy

   Compare two samples of vectors with the kernel maximum mean discrepancy: biased and unbiased estimators of MMD squared under a Gaussian kernel, the median-distance bandwidth, and a seeded permutation test.

   Runs with

   Python 3.10 or later

   Entry points

   mmd_squared, permutation_test, median_bandwidth

   Code moduleMIT10 files, 76.3 KBruns commands

7. Weighted Vendi score

   weighted_vendi_score

   Measure the effective number of distinct members in a weighted set from a similarity matrix or feature rows.

   Runs with

   Python 3.10 or later

   Entry points

   weighted_vendi_score, vendi_from_features

   Code moduleMIT10 files, 81.0 KBruns commands

8. Effective rank, participation ratio and stable rank

   effective_rank_spectral_entropy

   Measure how many directions a representation matrix actually uses: the effective rank (exponential of the entropy of normalized singular values), the participation ratio of the eigenvalues and the stable rank.

   Runs with

   Python 3.10 or later

   Entry points

   spectral_dimensions, singular_values

   Code moduleMIT10 files, 74.4 KBruns commands

9. Multicalibration audit by prediction bin and group

   multicalibration_audit

   Report the mean residual (outcome minus prediction) in every prediction bin for every declared group and for the whole population, flag cells beyond a tolerance, and repair violations by iterative patching.

   Runs with

   Python 3.10 or later

   Entry points

   multicalibration_audit, multicalibrate

   Code moduleMIT10 files, 84.0 KBruns commands

10. Within-group and between-group covariance decomposition

    correlation_source_decomposition

    Split the covariance between a score Y and a prediction P into the expected within-group covariance and the covariance of group means, exactly, using rational arithmetic.

    Runs with

    Python 3.10 or later

    Entry points

    decompose_covariance, decompose_variance

    Code moduleMIT10 files, 78.1 KBruns commands

11. Covariance intersection fusion

    covariance_intersection_fusion

    Fuse two mean and covariance estimates whose errors may be correlated in an unknown way, choosing the weight that minimizes the trace or determinant of the fused covariance.

    Runs with

    Python 3.10 or later

    Entry points

    covariance_intersection, independent_fusion, conservative_margin

    Code moduleMIT10 files, 84.5 KBruns commands

12. Dithered sign expectation and its inverse

    dithered_sign_expectation

    Compute the mean of sign(x + noise) for Gaussian or uniform dither, 2 Phi(x/sigma) - 1 or x/a, check it against a seeded Monte Carlo estimate with its standard error, and invert it to recover x from an observed mean of one-bit readings.

    Runs with

    Python 3.10 or later

    Entry points

    sign_expectation, monte_carlo_sign_expectation, recover_offset

    Code moduleMIT10 files, 73.4 KBruns commands

13. Transient amplification profile of a linear update

    transient_amplification_profile

    Compute ||A^h||_2 over a horizon for a linear update x_(h+1) = A x_h, with the peak, the step after which the norm stays below 1, the spectral radius and the Gelfand bound.

    Runs with

    Python 3.10 or later

    Entry points

    transient_profile, spectral_norm, spectral_radius

    Code moduleMIT10 files, 81.5 KBruns commands

14. Hodge decomposition of edge flows

    hodge_edge_flow_decomposition

    Split a flow on the edges of a small graph (pairwise preferences, score differences, transfers) into a gradient explained by a global potential, curl circulating around filled triangles, and harmonic circulation around holes.

    Runs with

    Python 3.10 or later

    Entry points

    hodge_decomposition, flow_from_potential

    Code moduleMIT10 files, 83.4 KBruns commands

15. Delta-rule associative memory

    delta_rule_associative_memory

    Store key-value pairs in a matrix with the error-correcting update M <- M + rate (v - M k) k^T, retrieve with M k, and compare with the Hebbian rule and the minimum-norm memory.

    Runs with

    Python 3.10 or later

    Entry points

    delta_update, hebbian_update, retrieve, store_pairs, least_squares_memory

    Code moduleMIT10 files, 79.3 KBruns commands