gallery.CohortForecast
One model, one cohort, one cutoff, one field, at the held-out cells.
Usage
gallery.CohortForecast(
model,
task,
cells,
field,
log_density=None,
draws=None,
density_absence=None,
draws_absence=None
)This granularity is forced from both sides. Below it, every layer is single-cohort and refuses to be otherwise: next_diagonal raises on more than one segment combination (measured before the guard: 20 January-origin cohorts plus one July-origin cohort returned 2 cells instead of 62), and all three contract builders raise on a multi-cohort triangle. Above it, every layer loops per cohort, and kernels.harness runs those loops in spawned processes - so whatever a worker returns must be picklable, which a live entry or an InferenceData is not. Plain numpy plus a HoldoutCells is.
cells is carried whole rather than copied piecemeal. That is provenance: only next_diagonal can build one, so a CohortForecast cannot describe cells that were never shown to be held out, and as_of, eval_date, the segment schema, the triangle basis, train_origins and the upstream exclusion census all arrive consistent with each other by construction rather than by five parallel arguments that can drift.
log_density is (n_draws, n_cells) on Lebesgue-on-the-loss-amount - the scale ScoresHeldout.log_lik_at guarantees and the only scale this module accepts. Either it or absence is set, never both and never neither.
A pooled fit (the NN entries fit once globally, then predict per cohort) produces one CohortForecast per scored cohort from a single fit. That is still held out - the pooled model trained on the as_of slice of every cohort - but model names the entry, not the fit, and any future clustered standard error must cluster on the FIT, which for those entries is the whole panel and not the cohort.
Refused at construction, each because the alternative computes:
- an array with no Absence, or an Absence with an array;
- a width that disagrees with the field-narrowed cell count - the check that pins the arrays to their keys;
- fewer than 2 draws (see
logmeanexp()); - zero cells: a cohort whose whole diagonal was excluded upstream has nothing to score, and a zero-width forecast would pass alignment invisibly while making the model look like it covered the cohort. Read
cells.exclusion_counts()instead; - any NaN;
- zero variance across draws in every cell, which means someone handed a repeated point estimate rather than posterior draws.
logmeanexpthen degenerates to the plug-in log density, which is systematically overconfident and looks completely normal.
Parameter Attributes
model: strtask: strcells: HoldoutCellsfield: strlog_density: np.ndarray | None = Nonedraws: np.ndarray | None = Nonedensity_absence: Absence | None = Nonedraws_absence: Absence | None = None
Attributes
| Name | Description |
|---|---|
| cohort |
The cohort’s segment VALUES. () for an unsegmented triangle.
|
| key_frame | One row per covered cell: the panel key, then the payload. |
| keys | Panel keys, in array column order. Tuples, so they set-operate. |
| measure | The TRIANGLE’s basis, cumulative or incremental - not the density’s. |
| n_draws |
Draw count behind whichever arrays are present, or None.
|
cohort
The cohort’s segment VALUES. () for an unsegmented triangle.
cohort: tuple
key_frame
One row per covered cell: the panel key, then the payload.
key_frame: pd.DataFrame
In the same order as log_density’s columns, which is what makes pointwise_elpd() joinable.
keys
Panel keys, in array column order. Tuples, so they set-operate.
keys: list[tuple]
measure
The TRIANGLE’s basis, cumulative or incremental - not the density’s.
measure: str
n_draws
Draw count behind whichever arrays are present, or None.
n_draws: int | None
The two capabilities may carry different draw counts - a density is one value per posterior draw while a sampler may thin - so this reports the density’s when there is one and the draws’ otherwise, and n_draws_for() answers per axis.
Methods
| Name | Description |
|---|---|
| ess_kish() |
(n_cells,) effective draws behind each cell. See _ess_kish().
|
| pointwise_crps() |
(n_cells,) CRPS per cell, against the realized outcomes. Lower is better.
|
| pointwise_elpd() |
(n_cells,) log predictive density per cell, logmeanexp over draws.
|
| unavailable() | A forecast that offers NEITHER capability, naming a reason for each. |
ess_kish()
(n_cells,) effective draws behind each cell. See _ess_kish().
Usage
ess_kish()pointwise_crps()
(n_cells,) CRPS per cell, against the realized outcomes. Lower is better.
Usage
pointwise_crps()The draws arrive on the TRIANGLE’s basis - that is PredictsHeldout.predict_at’s guarantee - so they are scored against cells.values directly and no scale question survives to this layer.
CRPS is translation-equivariant, so an entry that models increments and one that models cumulatives give comparable numbers once each is on its triangle’s basis; it is scale-dependent in the other sense, though, so a CRPS in dollars and one in thousands are not comparable. That is the same units question the panel’s value agreement check already catches, one layer up.
pointwise_elpd()
(n_cells,) log predictive density per cell, logmeanexp over draws.
Usage
pointwise_elpd()Per cell, THEN summed - never summed inside the exponential. Summing the log densities of a cohort’s cells before the reduction gives the log density of the whole diagonal jointly under each draw, which is a legitimate but different quantity: the cells share one posterior, so the joint is not the product of the marginals and it is not additive across cohorts in the way a leaderboard and a stacking objective both need.