Model metrics — model_metrics • prospicio

family_deviance() is sum(w d(y, mu)) for a family; gini_index() the Gini index of the ordered Lorenz curve (rows sorted by prediction, exposure share against loss share); lift_table() cuts rows sorted by predicted rate into bands of about equal exposure; crps_draws() the continuous ranked probability score of draws for an outcome; pinball_loss() the weighted mean pinball (quantile) loss of predicted alpha quantiles, sum(w * u * (alpha - (u < 0))) / sum(w) with u = y - pred, lowest in expectation at the true quantile.

Usage

family_deviance(family, y, mu, weights = NULL, theta = NULL, power = NULL)

gini_index(y, pred, exposure = NULL)

lift_table(y, pred, exposure = NULL, bands = 10)

crps_draws(draws, y)

pinball_loss(y, pred, alpha, weights = NULL)

log_score(
  family,
  y,
  mu,
  dispersion = 1,
  weights = NULL,
  theta = NULL,
  power = NULL
)

pit_values(
  family,
  y,
  mu,
  dispersion = 1,
  weights = NULL,
  seed = 0,
  theta = NULL,
  power = NULL
)

ks_uniform(values)

Arguments

family: A family name, as in glm_fit().
y: Outcomes.
mu, pred: Predictions.
weights, exposure: Optional weights or exposures.
theta, power: Family parameters, as in glm_fit().
bands: Number of lift bands.
draws: Equally likely draws.
alpha: Quantile level in (0, 1).
dispersion: The family's dispersion.
seed: Seed of the PIT's randomization.
values: Values to compare with the uniform.

Value

A number; for pit_values() one value per outcome; for lift_table() a data frame with columns exposure, expected and actual.

Details

log_score() is the mean of -log f(y) under each row's predictive distribution (the family with mean mu, dispersion and weight); pit_values() the probability integral transform F(y), randomized where the distribution has atoms (counts, a Tweedie's zero) and uniform when the model is calibrated; ks_uniform() the Kolmogorov-Smirnov distance of values from the uniform, about 1.36 / sqrt(n) or less 95% of the time under uniformity.

Examples

family_deviance("poisson", c(1, 0, 3), c(1, 0.5, 2))
#> [1] 1.432791
gini_index(c(0, 1), c(0.1, 0.9))
#> [1] 0.5
lift_table(c(0, 1, 2, 3), c(0.1, 0.9, 2.1, 2.9), bands = 2)
#>   exposure expected actual
#> 1        2        1      1
#> 2        2        5      5
crps_draws(c(1, 2, 3), 2)
#> [1] 0.2222222
pinball_loss(c(1, 0), c(0, 1), 0.9)
#> [1] 0.5
log_score("poisson", 0, 1)
#> [1] 1
set.seed(1)
y <- rpois(500, 3)
ks_uniform(pit_values("poisson", y, rep(3, 500)))
#> [1] 0.0319566