Author Details

Carlo Acerbi

Carlo Acerbi
Managing Director and Head of Risk Management Research

Social Sharing

Extended Viewer

BACKTESTING RISK MODELS: WHAT WE LEARNED

Risk measures, such as Expected Shortfall and Value at Risk, are designed to calculate the risk of a portfolio. But different risk models may work better than others for different asset classes and in varying time horizons. The MSCI Model Scorecard provides an innovative tool designed to help select the best risk model in terms of Expected Shortfall (ES) and Value at Risk (VaR) predictivity.

Results for the 12-month period ending in July 2015 indicate that the Monte Carlo and filtered historical models tended to perform better for fixed-income portfolios, whereas the historical model (with a relatively short lookback of 12 months) as well as the filtered and weighted historical models performed better for equity portfolios. However, it was difficult to generalize each model’s performance over different portfolios.

Besides ranking models in the MSCI Model Scorecard, we performed a traditional VaR backtest, by counting the number of times the realized loss of the portfolio exceeded the VaR forecast. A model which has too many “VaR exceedances” underestimates risk, a model with too few exceedances overestimates risk. This analysis was complemented by a number of conditional backtesting statistics, which were designed to detect inappropriate clustering of VaR exceedances.  

In addition to the traditional VaR backtest, this report includes a formal backtest of Expected Shortfall based on a framework recently developed by MSCI. Also, we validated the entire forecast distribution through the realized p-values.

We performed the above analysis for the 12-month period ending in July 2015 to examine how four types of simulation models available in RiskMetrics RiskManager behaved over the preceding 12 months: Monte Carlo, historical, filtered historical and weighted historical. The models were tested on portfolios replicating 10 indexes, representing different segments of U.S. and global equity and bond markets.

In addition to ranking the models for fixed-income and equity indexes, our July 2015 model backtests found:

  • When the risk models failed the VaR and ES backtests, this was usually the result of either a mild underestimation of risk (yellow zone in exhibit) or a mild overestimation of risk (light blue zone). Severe under- or over-estimation of risk was observed in only three instances.
  • Generally, all models passed the conditional backtests. A few exceptions were observed for the European fixed-income indexes, which went through turbulent periods in October 2014 and May-June 2015 due to economic uncertainty and a potential Greek exit from the eurozone. For these, the historical models and the Weighted Historical (decay factor 0.995) model revealed excessive clustering of times when realized losses exceeded VaR estimates.
  • The historical model with a five-year lookback period performed well in certain instances, but generally overestimated risk due to its long time horizon. The model also failed to capture the entire return distribution of the equity indexes.

 

Backtesting Statistics for 97.5% Expected Shortfall

 

Read the "Semi-annual report with detailed backtest results."

Pagination Portlet

Regulation