Scoring functions are an essential tool to evaluate point forecasts, and scoring rules to evaluate probabilistic forecasts. We start by reviewing some recent results on the construction of scoring functions and scoring rules. Point forecasts are issued on the basis of certain information. If the forecasting mechanisms are correctly specified, a larger amount of available information should lead to better forecasts. We show how the effect of increasing the information set on the forecast can be quantified by using strictly consistent scoring functions, and also discuss the role of the information set for evaluating probabilistic forecasts by using strictly proper scoring rules. Further, a method is proposed to test whether an increase in a sequence of information sets leads to distinct, improved $h$-step point forecasts. For the value at risk (VaR), we show that increasing the information set will result in VaR forecasts which lead to smaller expected shortfalls, unless an increase in the information set does not change the VaR forecast. The effect is illustrated in simulations and applications to stock returns for unconditional versus conditional risk management as well as univariate modeling of portfolio returns versus multivariate modeling of individual risk factors.
Reference: Holzmann, H., Eulert, M. (2014) The role of the information set for forecasting -- with applications to risk management. Annals of Applied Statistics 8, 595-621