- Artificial-intelligence techniques may be well-suited to model high-dimensional and highly nonlinear risk factors in the prepayment of agency mortgage-backed securities.
- MSCI’s AI agency prepayment model produced more accurate forecasts than traditional modeling techniques across historical periods and risk factors, during the period of our analysis.
- To avoid model risk, AI modelers and users can test a model’s risk-factor sensitivities against economic intuition.
Machine learning using neural networks has been successfully applied to many fields in which extremely complex patterns can prove challenging for other algorithms. We have argued that neural networks may be suited for many of the difficulties associated with modeling prepayment risk in mortgage-backed securities (MBS) — for example, vast amounts of data, large numbers of risk factors and the nonlinear and highly interactive nature of the risk factors. We applied feedforward neural networks on pool-level agency-MBS data to build an AI prepayment model.1
But how well did the AI model accomplish the task? Here, we pit the MSCI AI agency prepayment model against our production model (or the human model), which was built and updated using traditional modeling techniques, often involving painstakingly iterative processes.
The AI Model’s Greater Accuracy in Modeling Prepayment
The exhibit below illustrates the AI model’s relative success in differentiating prepayment behaviors between California and New York pools, which have different attributes — such as loan size and credit profile. More important, New York’s mortgage-recording tax has tended to lower prepayment response to refinancing opportunities. The AI model produced more accurate forecasts than our human model.
Model Error Tracking for Pools Concentrated in California and New York
In addition, the AI model accurately captured the general relationship between pool attributes like loan size, FICO score, loan-to-value ratios and prepayment speed.2 In backtesting, the exhibit below shows that the AI model produced better model forecasts than our human model, using our rank-based error-tracking method.3 Measured across prepayment-ranked percentile buckets, the AI model better differentiated the low- and high-speed buckets and was more accurate than our human model, during our analysis period: The average absolute error across the ranking buckets was 0.1 single monthly mortality (SMM) for the AI model — much smaller than the human model’s 0.3 SMM.
Rank-Based Error Tracking for the AI and Human Models
Modeling Media Effect
The exhibit below shows the AI model’s ability to capture extremely complex prepayment behavior, such as the “media effect,” in which media coverage of declining mortgage rates influences the trend of prepayment. Mortgages with higher coupons have tended to prepay faster due to their higher refinance incentives. But when rates reach historical lows and a large segment of the mortgage universe becomes refinanceable, newer and lower-coupon mortgages have tended to respond to refinancing incentives much more quickly and reach much higher prepayment speeds than mortgages with higher coupons. This phenomenon is caused by complex dynamics between borrowers and mortgage originators.4 The AI model was able to forecast this complex behavior more accurately than our human model.
Model Error Tracking in the 2012 Refinance Wave
Prepayment speeds for 3.5, 4 and 4.5s coupons of the 2012 vintage.
How do we know the small modeling error is the result of the model’s dynamics and not overfitting? Besides having used many specific AI techniques to avoid overfitting,5 we tested the results by examining the AI model’s behavior by varying the risk factors to verify the model dynamics were consistent with economic intuition. For example, the exhibit below shows the AI model’s prepayment response to the refinance incentive (the refinance “s-curve”), at different loan-size levels.
AI Model’s Refinance ‘S-Curve’ for Different Loan Sizes
Drilling down further into the nonlinear interaction between loan size and incentive, the exhibit below shows that discount-mortgage prepayment speeds decreased with loan size, while premium-mortgage prepayment speeds increased with loan size. The AI model’s dynamics were consistent with economic intuition.
Prepayment Speeds Decreased with Loan Size for Negative Incentives, Increased for Positive Ones
In sum, the AI model overcame the issues of high dimensionality and high nonlinearity in agency prepayment modeling during the period of analysis. With proper selection of modeling parameters, it reduced the length of the modeling process from months to hours.
1For details, see: Zhang, D., Zhao, X., Zhang, J., Teng, F., Siyu, L., and Li, H. 2019. “Agency MBS Prepayment Model Using Neural Networks.” Journal of Structured Finance.
3Zhang, J. 2019. “MSCI Rank-Based Error Tracking for Agency MBS Prepayment Models.” MSCI Model Insight. (Client access only.) This paper will appear in the Journal of Fixed Income in summer 2020.
4“Agency MBS Prepayment Model Using Neural Networks.”
MSCI Agency Prepayment Model Performance Review 2019 (client access only)
MSCI Rank-Based Error Tracking for Agency MBS Prepayment Models (client access only)