# Statistics for comparing results

Part of Deterministic skills by Baltor.

Compare two models, two prompts or two harness setups with statistics that hold up: calibration, conformal intervals, confidence sequences, covariance shrinkage and kernel two-sample tests.

## What to watch for

- A point estimate without an interval cannot say whether two runs differ.
- Checking a running experiment again and again inflates false positives unless the interval is anytime-valid.
- With many features and few rows a sample covariance is unstable, so shrink it before you invert it.

## Components

- [Brier score decomposition and expected calibration error](https://baltor.ai/components/brier-score-decomposition-and-expected-calibration-error-9e958209): Split the Brier score of probability forecasts into reliability, resolution, uncertainty and the two within-bin terms that make the decomposition exact for binned forecasts, and compute the expected calibration error. Name `brier_decomposition_calibration_error`, licence MIT, 10 files.
- [Split conformal prediction intervals](https://baltor.ai/components/split-conformal-prediction-intervals-a12faaa2): Turn any point predictor into prediction intervals with a finite-sample coverage guarantee, using held-out calibration residuals and the exact ceil((n + 1)(1 - alpha)) quantile index, optionally per group (Mondrian), with a seeded coverage check on held-out data. Name `split_conformal_intervals`, licence MIT, 10 files.
- [Anytime-valid confidence sequence for a bounded mean](https://baltor.ai/components/anytime-valid-confidence-sequence-for-a-bounded-mean-84f408d3): Give confidence intervals for the mean of bounded observations that hold at every time simultaneously, so results can be monitored after each run and stopped at any point. Name `anytime_confidence_sequence`, licence MIT, 10 files.
- [Probabilistic and deflated Sharpe ratio](https://baltor.ai/components/probabilistic-and-deflated-sharpe-ratio-e7d9db39): Compute the probability that a strategy's true Sharpe ratio exceeds a benchmark given skewness, kurtosis and sample length, and deflate it for having picked the best of many trials by comparing against the expected maximum Sharpe ratio of pure-luck trials. Name `deflated_sharpe_ratio`, licence MIT, 10 files.
- [Ledoit-Wolf covariance shrinkage](https://baltor.ai/components/ledoit-wolf-covariance-shrinkage-20713d30): Shrink a sample covariance toward a scaled identity with the Ledoit-Wolf intensity, giving a well-conditioned, positive definite estimate even when there are fewer observations than variables. Name `ledoit_wolf_shrinkage`, licence MIT, 10 files.
- [Maximum mean discrepancy with an RBF kernel](https://baltor.ai/components/maximum-mean-discrepancy-with-an-rbf-kernel-979b66d3): Compare two samples of vectors with the kernel maximum mean discrepancy: biased and unbiased estimators of MMD squared under a Gaussian kernel, the median-distance bandwidth, and a seeded permutation test. Name `rbf_maximum_mean_discrepancy`, licence MIT, 10 files.
- [Weighted Vendi score](https://baltor.ai/components/weighted-vendi-score-7f83db0b): Measure the effective number of distinct members in a weighted set from a similarity matrix or feature rows. Name `weighted_vendi_score`, licence MIT, 10 files.
- [Effective rank, participation ratio and stable rank](https://baltor.ai/components/effective-rank-participation-ratio-and-stable-rank-83a74966): Measure how many directions a representation matrix actually uses: the effective rank (exponential of the entropy of normalized singular values), the participation ratio of the eigenvalues and the stable rank. Name `effective_rank_spectral_entropy`, licence MIT, 10 files.
- [Multicalibration audit by prediction bin and group](https://baltor.ai/components/multicalibration-audit-by-prediction-bin-and-group-587989e0): Report the mean residual (outcome minus prediction) in every prediction bin for every declared group and for the whole population, flag cells beyond a tolerance, and repair violations by iterative patching. Name `multicalibration_audit`, licence MIT, 10 files.
- [Within-group and between-group covariance decomposition](https://baltor.ai/components/within-group-and-between-group-covariance-decomposition-10d84ed9): Split the covariance between a score Y and a prediction P into the expected within-group covariance and the covariance of group means, exactly, using rational arithmetic. Name `correlation_source_decomposition`, licence MIT, 10 files.
- [Covariance intersection fusion](https://baltor.ai/components/covariance-intersection-fusion-6f75e7c1): Fuse two mean and covariance estimates whose errors may be correlated in an unknown way, choosing the weight that minimizes the trace or determinant of the fused covariance. Name `covariance_intersection_fusion`, licence MIT, 10 files.
- [Dithered sign expectation and its inverse](https://baltor.ai/components/dithered-sign-expectation-and-its-inverse-48018416): Compute the mean of sign(x + noise) for Gaussian or uniform dither, 2 Phi(x/sigma) - 1 or x/a, check it against a seeded Monte Carlo estimate with its standard error, and invert it to recover x from an observed mean of one-bit readings. Name `dithered_sign_expectation`, licence MIT, 10 files.
- [Transient amplification profile of a linear update](https://baltor.ai/components/transient-amplification-profile-of-a-linear-update-047a4c5f): Compute ||A^h||_2 over a horizon for a linear update x_(h+1) = A x_h, with the peak, the step after which the norm stays below 1, the spectral radius and the Gelfand bound. Name `transient_amplification_profile`, licence MIT, 10 files.
- [Hodge decomposition of edge flows](https://baltor.ai/components/hodge-decomposition-of-edge-flows-9134093b): Split a flow on the edges of a small graph (pairwise preferences, score differences, transfers) into a gradient explained by a global potential, curl circulating around filled triangles, and harmonic circulation around holes. Name `hodge_edge_flow_decomposition`, licence MIT, 10 files.
- [Delta-rule associative memory](https://baltor.ai/components/delta-rule-associative-memory-b492f4e7): Store key-value pairs in a matrix with the error-correcting update M <- M + rate (v - M k) k^T, retrieve with M k, and compare with the Hebbian rule and the minimum-norm memory. Name `delta_rule_associative_memory`, licence MIT, 10 files.

Your agent downloads these through Baltor with Pro, $49 a month: https://baltor.ai/get-started