Clarify how to pass metrics to `score()`

[jamesmbaazam]

I was doing something like example_sample_continuous |> score(metrics = list("crps" = crps_sample()) instead of example_sample_continuous |> score(metrics = list("crps" = crps_sample) and getting the error r "Error in assert_numeric(observed, min.len = 1) : argument "observed" is missing, with no default". The main difference here is the () at the end of the first invocation. That's because the helpfile mentioned you need to pass a function. I think this can be clarified by adding an example in score(). It took me a lot of digging in the code to clarify this.

[nikosbosse]

Huh, also I guess this should just have been caught by the checks....

[jamesmbaazam]

A bit of this can be blamed on my comprehension skills.

[seabbs]

Is the docs for making a cutom metric list documented from the score function? That is the only place that this documented I think

[jamesmbaazam]

Those docs only talk about passing extra/non-default values to the metric arguments. From what I understood.

But as a user, how can I pass a subset of the inbuilt metric functions? I struggled to answer that from the help file and had to dig into the code to find apply_metrics().

[seabbs]

The answer is select_metrics: https://epiforecasts.io/scoringutils/reference/select_metrics.html?q=select#null but it doesn't have very good doc links to it

[seabbs]

Did you have any thoughts on what we could do here @jamesmbaazam?

[jamesmbaazam]

I think you can also use get_metrics(). The easiest fix here would be to provide more examples showing how to use a subset of the metrics. An examples is in the issue description. Other routes below.

library(scoringutils)
# Using get_metrics()
score(example_quantile, get_metrics(example_quantile, "bias"))
#>      location target_end_date target_type location_name forecast_date
#>        <char>          <Date>      <char>        <char>        <Date>
#>   1:       DE      2021-05-08       Cases       Germany    2021-05-03
#>   2:       DE      2021-05-08       Cases       Germany    2021-05-03
#>   3:       DE      2021-05-08       Cases       Germany    2021-05-03
#>   4:       DE      2021-05-08      Deaths       Germany    2021-05-03
#>   5:       DE      2021-05-08      Deaths       Germany    2021-05-03
#>  ---                                                                 
#> 883:       IT      2021-07-24      Deaths         Italy    2021-07-12
#> 884:       IT      2021-07-24      Deaths         Italy    2021-07-05
#> 885:       IT      2021-07-24      Deaths         Italy    2021-07-12
#> 886:       IT      2021-07-24      Deaths         Italy    2021-07-05
#> 887:       IT      2021-07-24      Deaths         Italy    2021-07-12
#>                      model horizon  bias
#>                     <char>   <num> <num>
#>   1: EuroCOVIDhub-ensemble       1  0.50
#>   2: EuroCOVIDhub-baseline       1  0.95
#>   3:  epiforecasts-EpiNow2       1  0.90
#>   4: EuroCOVIDhub-ensemble       1 -0.10
#>   5: EuroCOVIDhub-baseline       1  0.30
#>  ---                                    
#> 883: EuroCOVIDhub-baseline       2  0.20
#> 884:       UMass-MechBayes       3  0.10
#> 885:       UMass-MechBayes       2  0.80
#> 886:  epiforecasts-EpiNow2       3  0.50
#> 887:  epiforecasts-EpiNow2       2  0.90
# Using select_metrics() and get_metrics()
score(example_quantile, select_metrics(get_metrics(example_quantile), "bias"))
#>      location target_end_date target_type location_name forecast_date
#>        <char>          <Date>      <char>        <char>        <Date>
#>   1:       DE      2021-05-08       Cases       Germany    2021-05-03
#>   2:       DE      2021-05-08       Cases       Germany    2021-05-03
#>   3:       DE      2021-05-08       Cases       Germany    2021-05-03
#>   4:       DE      2021-05-08      Deaths       Germany    2021-05-03
#>   5:       DE      2021-05-08      Deaths       Germany    2021-05-03
#>  ---                                                                 
#> 883:       IT      2021-07-24      Deaths         Italy    2021-07-12
#> 884:       IT      2021-07-24      Deaths         Italy    2021-07-05
#> 885:       IT      2021-07-24      Deaths         Italy    2021-07-12
#> 886:       IT      2021-07-24      Deaths         Italy    2021-07-05
#> 887:       IT      2021-07-24      Deaths         Italy    2021-07-12
#>                      model horizon  bias
#>                     <char>   <num> <num>
#>   1: EuroCOVIDhub-ensemble       1  0.50
#>   2: EuroCOVIDhub-baseline       1  0.95
#>   3:  epiforecasts-EpiNow2       1  0.90
#>   4: EuroCOVIDhub-ensemble       1 -0.10
#>   5: EuroCOVIDhub-baseline       1  0.30
#>  ---                                    
#> 883: EuroCOVIDhub-baseline       2  0.20
#> 884:       UMass-MechBayes       3  0.10
#> 885:       UMass-MechBayes       2  0.80
#> 886:  epiforecasts-EpiNow2       3  0.50
#> 887:  epiforecasts-EpiNow2       2  0.90

^{Created on 2025-07-18 with reprex v2.1.1}