Finding the Sentiment Hidden in Regulatory Filings

Blog post

9 min read

February 24, 2021

We used natural-language processing to create a sentiment factor out of company regulatory filings by quantifying how much a company changed its filing and in what sections changes were made.
A hypothetical equal-weighted portfolio of companies that made the most changes to their regulatory filings (had low exposure to our reporting sentiment factor) strongly underperformed the market.
Our reporting sentiment factor had strong positive performance since 2009 and showed low correlation to traditional style factors throughout the study period.

Regulatory filings constitute one of the most important ways for investors and other stakeholders to learn about the business and financial state of a publicly traded company, and large changes in company filings may indicate new material information. Using quarterly (10-Q) and annual (10-K) SEC filings of companies included in the MSCI USA Investable Market Index (IMI) and natural-language processing (NLP), we defined and calculated a number of document metrics that may have explained a portion of stock returns. Our approach, such as those detailed in prior literature,¹ measures changes in company filings by computing metrics such as textual and structural document similarity and changes in document size.

Defining Document Metrics and Our Reporting Sentiment Factor

We use a range of NLP techniques to define and calculate document metrics of a company's regulatory filings. We start with making the SEC filings less noisy by processing the text to remove the most common words in the English language² and focusing on root forms of words.³ We then compute textual and structural similarity of filings by comparing the word-embedding⁴ representations of equivalent texts in two filings. We compute the "document similarity" metric,⁵ which measures similarity based on word frequency and contextual usage inside the filings. We also compute the "change in document size" metric using a count of words on a preprocessed filing. We generate these metrics for 10-Ks by comparing two consecutive 10-K filings and for 10-Qs on a quarter-over-quarter (QOQ) and year-over-year (YOY) basis.⁶ In addition, we compute metrics separately for whole filings and for individual sections, such as "Management's Discussion & Analysis" and "Legal Proceedings," that may be more likely to cover new material and unique information related to the business of a company. Last, we define our reporting sentiment factor as an equal-weighted combination of exposure z-scores⁷ of component descriptors: integrated document similarity and integrated change in document size. In the table below, we define the descriptors.

Descriptor	Definition
Integrated Document Similarity	Equal-weighted combination of exposure z-scores of the "document similarity" metric computed for: 10-K whole filing, 10-K "Management's Discussion & Analysis," 10-K "Legal Proceedings," 10-Q whole filing (YOY), 10-Q "Management's Discussion & Analysis" (YOY), 10-Q whole filing (QOQ), 10-Q "Management's Discussion & Analysis" (QOQ)
None	Equal-weighted combination of exposure z-scores of the "document similarity" metric computed for: 10-K whole filing, 10-K "Management's Discussion & Analysis," 10-K "Legal Proceedings," 10-Q whole filing (YOY), 10-Q "Management's Discussion & Analysis" (YOY), 10-Q whole filing (QOQ), 10-Q "Management's Discussion & Analysis" (QOQ)
Integrated Change in Document Size	Equal-weighted combination of exposure z-scores of the "change in document size" metric computed for: 10-K whole filing, 10-K "Management's Discussion & Analysis," 10-K "Legal Proceedings," 10-Q whole filing (YOY), 10-Q "Management's Discussion & Analysis" (YOY), 10-Q Whole filing (QOQ), 10-Q "Management's Discussion & Analysis" (QOQ)

For both descriptors, we take the "Legal Proceedings" section for 10-K filings only, because companies cover this in greater depth in 10-K filings than in 10-Qs.

None

Equal-weighted combination of exposure z-scores of the "document similarity" metric computed for: 10-K whole filing, 10-K "Management's Discussion & Analysis," 10-K "Legal Proceedings," 10-Q whole filing (YOY), 10-Q "Management's Discussion & Analysis" (YOY), 10-Q whole filing (QOQ), 10-Q "Management's Discussion & Analysis" (QOQ)

Stock Performance and Size of Companies with Large Changes in Filings

To analyze the relationship between large changes in company filings and subsequent stock performance, we divided the constituents of the MSCI USA IMI into deciles based on the highest to the lowest amount of changes in regulatory filings. We then analyzed performance by equally weighting constituents of the deciles over one- and three-month periods. The exhibit below shows that the deciles with large changes in regulatory filings — low exposure to the reporting sentiment factor — strongly underperformed the market, whereas the deciles with small changes in filings — high exposure to the reporting sentiment factor — marginally outperformed the market.

Companies with Low Exposure to the Reporting Sentiment Factor Typically Underperformed

The 10 different deciles are constructed by sorting the MSCI USA IMI universe by the lowest to the highest reporting sentiment score (highest to the lowest change in filings). The deciles are rebalanced with one-month and three-month rebalancing frequencies, and the annual average return is the equal-weighted return of securities in these deciles. The period of analysis is between 2007 and 2020.

We also computed active-size-factor exposure of these deciles to find the market-capitalization size distribution of companies with varying degree of changes in filings. The exhibit below shows that companies with large amounts of changes in regulatory filings — low exposure to the reporting sentiment factor — tended to be smaller compared to other companies in the MSCI USA IMI.

Companies with Low Reporting Sentiment-Factor Exposure Typically Were Smaller

The active size exposure is relative to the size exposure of the equal-weighted MSCI USA IMI. The deciles are constructed by sorting the MSCI USA IMI universe by the lowest to the highest reporting sentiment score (highest to the lowest change in filings). The deciles are rebalanced on a monthly basis. The constituents of deciles are weighted equally, and the period of analysis is between 2007 and 2020.

Reporting Sentiment Factor's Descriptors Captured Unique Components of Filing Changes

We computed performance of the reporting sentiment factor and its component descriptors by introducing each as an additional equity style factor in multivariate cross-sectional regression in the MSCI US Total Market Equity Trading Model (USFAST). This method helps generate the unique potential contribution of a given factor over and above all other risk-model factors. The exhibit below shows that the integrated document similarity and integrated change in document-size descriptors had positive performance on a pure-factor and decile-spread basis. It also shows that the reporting sentiment factor outperformed its component descriptors, suggesting the descriptors captured different dimensions of changes in filings. The time-series performance chart shows that the performance of the reporting sentiment factor improved from 2009 onward, perhaps due to the better quality of financial reporting mandated by the SEC after the 2008 global financial crisis.⁸ In addition, lower t-stat and lower factor volatility are consistent with the characteristics of alpha factors versus those of risk factors. Last, the high-exposure auto-correlation, as a result of a quarterly factor-exposure-calculation frequency, suggests the factor may be suitable for medium- to long-term investment horizons.⁹

Performance Characteristics of Reporting Sentiment Factor and Its Components

None	Decile portfolio statistics	Decile portfolio statistics	Decile portfolio statistics	Multivariate regression statistics	Multivariate regression statistics	Multivariate regression statistics	Multivariate regression statistics	Multivariate regression statistics	Stability
None	Decile spread return ,%	Decile Spread volatility ,%	Decile spread IR	Factor return ,%	Factor volatility ,%	Factor IR	Mean \|t\|	\|t\| > 2, %	Exposure auto correct
Reporting Sentiment	3.06	6.4	0.48	1.03	0.87	1.18	1.06	11.38	0.82
Reporting Sentiment	None	None	None	None	None	None	None	None	None
3.06	None	None	None	None	None	None	None	None	None
6.4	None	None	None	None	None	None	None	None	None
0.48	None	None	None	None	None	None	None	None	None
1.03	None	None	None	None	None	None	None	None	None
0.87	None	None	None	None	None	None	None	None	None
1.18	None	None	None	None	None	None	None	None	None
1.06	None	None	None	None	None	None	None	None	None
11.38	None	None	None	None	None	None	None	None	None
0.82	None	None	None	None	None	None	None	None	None
Integrated Document Similarity	0.73	7.45	0.1	0.65	0.88	0.74	1.05	11.38	0.93
Integrated Document Similarity	None	None	None	None	None	None	None	None	None
0.73	None	None	None	None	None	None	None	None	None
7.45	None	None	None	None	None	None	None	None	None
0.1	None	None	None	None	None	None	None	None	None
0.65	None	None	None	None	None	None	None	None	None
0.88	None	None	None	None	None	None	None	None	None
0.74	None	None	None	None	None	None	None	None	None
1.05	None	None	None	None	None	None	None	None	None
11.38	None	None	None	None	None	None	None	None	None
0.93	None	None	None	None	None	None	None	None	None
Integrated Change in Document Size	2.37	5.21	0.45	0.76	0.73	1.04	0.98	8.39	0.7
Integrated Change in Document Size	None	None	None	None	None	None	None	None	None
2.37	None	None	None	None	None	None	None	None	None
5.21	None	None	None	None	None	None	None	None	None
0.45	None	None	None	None	None	None	None	None	None
0.76	None	None	None	None	None	None	None	None	None
0.73	None	None	None	None	None	None	None	None	None
1.04	None	None	None	None	None	None	None	None	None
0.98	None	None	None	None	None	None	None	None	None
8.39	None	None	None	None	None	None	None	None	None

Reporting Sentiment

3.06

6.4

0.48

1.03

0.87

1.18

1.06

11.38

0.82

Integrated Document Similarity

0.73

7.45

0.1

0.65

0.88

0.74

1.05

11.38

0.93

Integrated Change in Document Size

2.37

5.21

0.45

0.76

0.73

1.04

0.98

8.39

Factor return is computed via multivariate cross-sectional regression at monthly frequency. Decile spread is also computed at a monthly frequency. Cumulative return is the arithmetic cumulative sum of monthly returns. Exposure auto-correlation is an average of month-over-month cross-sectional exposure correlation of the factor z-scores. The period of analysis is between 2007 and 2020.

Reporting Sentiment Factor Had Low Correlation with Traditional Style Factors

We also analyzed cross-sectional exposure correlation of the reporting sentiment factor with other style factors in USFAST. The exhibit below shows that the factor, during certain periods, had marginally high correlation to the earnings quality, management-quality, size, earnings-yield, leverage and residual-volatility factors, between 2007 and 2020, suggesting that companies with large changes in regulatory filings were less stable, had higher leverage and lower size (as shown earlier). On average, however, the factor had low cross-sectional exposure correlation to the existing style factors in USFAST, over the study period.

Cross-Sectional Exposure Correlation of Reporting Sentiment with Traditional Style Factors

Exposure correlation is computed on a monthly basis by measuring cross-sectional exposure correlation between the reporting-sentiment factor and equity style factors in USFAST. The computation is done using the MSCI USA IMI universe. The period of analysis is between 2007 and 2020.

Persistence of Factor Information for the Reporting Sentiment Factor

Finally, we analyzed how long information in the factor reflected subsequent stock performance. For this, we computed the cross-sectional rank information coefficient between the factor score and the subsequent stock return over different periods. The exhibit below shows that the information in the factor continued to accrue for a long period and did not reverse, implying that far from overreaction in the short term, changes in filings may reflect true fundamental information for firms, which gets gradually incorporated into the stock price in the months after the reporting change.

Reporting Sentiment Factor's Information Content Remained Intact over Longer Periods

Exposure correlation is computed on a monthly basis by measuring cross-sectional exposure correlation between the reporting sentiment factor and equity style factors in USFAST. The computation is done using the MSCI USA IMI universe. The period of analysis is between 2007 and 2020. Many dimensions of sentiment can be estimated from sources for structured financial-market data. Using NLP techniques on regulatory filings, we constructed a sentiment factor that quantifies changes in the tone and content of company filings, and which may be an indicator of future risks facing a company. We found that between 2007 and 2020, the factor generated positive performance and had low correlation to traditional style factors. We believe a reporting sentiment factor is a promising avenue for investigation.¹See, for Example:
Brown, Steven and Wu Tucker, Jennifer. 2011. "Large‐Sample Evidence on Firms' Year‐over‐Year MD&A Modifications." Journal of Accounting Research. 49 (2): 309-346; Cohen, L., Malloy, C., and Nguyen, K. 2020. "Lazy Prices." The Journal of Finance

Subscribe today
to have insights delivered to your inbox.

² We use the “stop-words removal” NLP technique, which removes the most common words in a language.³ We use “stemming and lemmatization” NLP techniques to generate the root form of the inflected words. The difference is that stem might not be an actual word, whereas the lemma is an actual word.⁴ Word embeddings are language-modeling techniques in NLP where words or phrases from a text “corpus” (group of documents) are mapped to numerical vectors representing related and co-occurring words or to vectors of linguistic contexts in which the words occur⁵ The document-similarity metric measures cosine similarity between the word embeddings of two filings, using the TF-IDF and Doc2Vec word-embedding models.⁶ YOY is defined as comparison between the current quarter’s filing and the filing of the same quarter in the previous year.⁷Z-scores are computed by standardizing the raw descriptor values, so that each descriptor has a market-capitalization-weighted mean of zero and a unit standard deviation.⁸ “Final Rule: Interactive Data to Improve Financial Reporting.” Securities and Exchange Commission, Jan. 30, 2009.⁹ Exposure auto-correlation above .9 would typically represent exposure half-life of six-months, whereas that between .5 and .9 would represent exposure half-life greater than one month. Factors with such exposure auto-correlation have been more suitable for medium- to long-term investing.

The content of this page is for informational purposes only and is intended for institutional professionals with the analytical resources and tools necessary to interpret any performance information. Nothing herein is intended to recommend any product, tool or service. For all references to laws, rules or regulations, please note that the information is provided “as is” and does not constitute legal advice or any binding interpretation. Any approach to comply with regulatory or policy initiatives should be discussed with your own legal counsel and/or the relevant competent authority, as needed.

Featured solutions bring together a variety of products to create approaches customized to your needs

MSCI Research Weekly

By topic

By asset class

Geospatial Asset Intelligence

Index categories

Index resources

MSCI All Country Venture-Backed Private Company Indexes

By asset class

By theme

Understanding market disruption

Who we are

News and events

Innovation

Grow with us

Finding the Sentiment Hidden in Regulatory Filings

Subscribe today
to have insights delivered to your inbox.

Featured solutions bring together a variety of products to create approaches customized to your needs

MSCI Research Weekly

By topic

By asset class

Geospatial Asset Intelligence

Index categories

Index resources

MSCI All Country Venture-Backed Private Company Indexes

By asset class

By theme

Understanding market disruption

Who we are

News and events

Innovation

Grow with us

Finding the Sentiment Hidden in Regulatory Filings

Subscribe todayto have insights delivered to your inbox.

Subscribe today
to have insights delivered to your inbox.