gallery.align_panel()

Put every model on the SAME cells - per score - and report what fell out.

Usage

Source

gallery.align_panel(
    forecasts,
    *,
    units=None,
)

Each score’s panel is the intersection over the models eligible for that score: a cell survives on the ELPD panel only if every density-member offered a density for it, and on the CRPS panel only if every draws-member offered draws for it. That is not a convenience, it is what makes either column a comparison at all - two totals over different cell sets differ by the cells, not by the models, and that difference is invisible on the board because both numbers are finite, the right sign and the right order of magnitude. Normalizing per cell does not repair it either: a cell’s log density is -log(y_i) plus a model term of order 1, so a per-cell mean over a different set of cohorts is dominated by cohort SIZE.

The two panels are separate, and that is load-bearing rather than fastidious. england_verrall_odp is a draws-member and not a density-member; if there were one shared panel, its refusing about half the mart would delete cells from meyers_csr’s ELPD - a column ODP does not appear in at all.

Membership, and what it costs. A model is a member for a score if it has no model-level absence on that axis and offered that capability for at least one cohort. So:

  • deterministic/mack is a CRPS member and never an ELPD one: its bootstrap draws fine but states no observation model. england_verrall_odp, clark_growth_curve and statistical/clark are the same shape.
  • A member that refuses one cohort removes that cohort from that score’s panel for every other member of that score. The price is real and it is reported cell by cell in dropped, which names both the score and the members that lacked the cell; the alternative is an incomparable column.
  • A model that offered nothing on an axis is excluded from that membership rather than allowed to empty the panel, and its row says na: no_scored_cohorts. Including it would collapse that column to zero cells and tell the reader nothing about anyone.

task, as_of, the segment schema and the triangle basis are DERIVED from the forecasts and a mix is refused, rather than being parameters. The precedent is index_into, where field was made to default to the contract’s own: a caller who can pass the task is a caller who can pass the wrong one, and the forecasts already know.

Raises, each because the alternative computes:

mixed task The paid and reported boards stay separate. Merging them sums densities of two different quantities into one number that still prints. mixed as_of Two cutoffs is two panels. See ForecastPanel. mixed segment schema A cohort keyed on (company, line) and one keyed on line alone cannot be told apart along the missing dimension - the same reason index_into compares schemas and not just values. duplicate (model, cell) Two forecasts of one model covering one cell, e.g. a re-run appended to a results list. The cell would count twice in that model’s sum and once in everyone else’s. disagreeing observed value Two models report different outcomes at the same key. Reachable, and the likeliest cause is not exotic: Meyers’ pmax(paid, 1) clamp, which the retro harness applies before fitting for the lognormal entries only. A clamped run and an unclamped one produce identical keys and different values at exactly the cells that matter. It is also what catches a units mismatch, which would otherwise shift every log density by a constant large enough to decide the ranking. disagreeing eval_date Same cohort and cutoff, different next diagonal: the models were handed different training slices, so nothing below is a comparison. disagreeing measure One model’s cells are cumulative and another’s incremental. The densities would be comparable (Jacobian 1) but the observed values are different numbers. disagreeing train_origins The models were fitted over different origin windows. This passes every other check here - same keys, same values, same eval_date - and CLAUDE.md already records this exact bug class inflating an outcome aggregate 2.4x. disagreeing exclusion counts The upstream held-out definitions differ, i.e. the models were not asked the same question. empty intersection Raised WITH the per-model coverage counts, so the caller can see which member emptied the board and drop it deliberately. Returning an empty panel would let a leaderboard of zeros be published.

An entry whose contract carries delta (compartmental) reaches here like any other: training_index returns a DeltaCellIndex for it, so its scorer has the same in-sample agreement gate, and its board forecasts cover the paid field only - ‘outstanding’ is derived, not a raw field a panel can intersect on.