This is Part 9 in a series about smart beta ETFs. See below for links to other posts in the series. In this installment, we look at how you can tell whether smart beta indexes will really perform as expected.

 

As we’ve worked through each of the factors targeted by smart beta ETFs—value, size, momentum, low volatility and quality—we’ve been careful to point out that no one really knows whether these premiums will persist in the future, especially once they’re on the radar of millions of investors.

However, we can look backward to see whether smart beta indexes have actually behaved as you would expect. Say you’re considering an ETF that targets value and small-cap stocks. If the index did well when value and small stocks outperformed, and did poorly when they lagged, that’s reassuring. But if you find the index’s performance had no correlation with value and size—or if it delivered outsized returns during periods when these stocks were dogs—that’s a red flag. It would be difficult to put a lot of faith in that ETF.

Think of a gardener who buys a system of rain barrels to capture the precipitation that falls on her roof. She knows she can never predict the weather next month, but if she can be confident the barrels will reliably capture a high percentage of any rain that does fall, that would be enough to justify the purchase. And if she finds the barrels full after two weeks of drought, that’s cause for concern, because she can’t explain where that water came from.

Turns out there is a way to put smart beta strategies to the test: it’s a statistical tool called regression analysis. Justin Bender has described the process in detail, but we can skip the gory details here and focus on the bigger picture. The general idea is to obtain the monthly performance data for each factor (value, size, quality and so on), according to its “academic” definition. Then you get monthly performance data for the fund you’re interested in, or its index. Finally, you compare these sets of data to see whether the strategy performed as you would expect.

Blitzing the data

Some of the most interesting work in this area comes from Dutch researcher David Blitz, summarized in a recent paper called Factor Investing with Smart Beta Indices. Using the method I’ve described above, Blitz examined popular smart beta indexes from Russell, MSCI and S&P to see whether they really captured the factors they targeted.

The results were mixed. The well-known Russell 1000 Value Index, for example, turned out to be “not very suitable for investors seeking pure and sizable exposure to the (academic) value factor.” Meanwhile, “the MSCI High Dividend index provides a huge (83%) exposure to the Low Volatility factor,” which suggests that high dividends are not really driving the returns of that strategy. There was water in the barrels, but it didn’t come from rainfall.

Overall, Blitz argued that investors will have a tough time selecting smart beta ETFs from the growing menu of choices. “The amount of factor exposure provided by popular smart beta strategies differs considerably,” he writes, and “these results imply that factor investing with smart beta indices is not as straightforward as one might think.”

The first dimension

Before we look at some specific smart beta strategies in this light, let’s discuss how to interpret the data. By definition, a traditional index fund has no additional exposure to any of the factors: it’s all “dumb beta.” So if you did a regression analysis on a plain-vanilla index ETF you should expect to get a value of 1 for the market factor (beta) and a zero for all the others. This would indicate that 100% of the fund’s performance can be explained by its exposure to the broad market:

Theoretical Broad-Market Index Fund
Market (beta) 1.00
Value 0.00
Size 0.00
Momentum 0.00
Quality 0.00

A good place to start our examination of smart beta strategies is Dimensional Fund Advisors, one of the pioneers in this area. Although DFA doesn’t offer ETFs, their mutual funds are designed to get “strong exposure to securities of small issuers and securities that it considers to be value securities.” They now also include an additional screen for companies with high profitability, which is closely related to what we’ve been calling the quality factor.

I asked Justin to run a regression on these indexes to see how much of their performance could be explained by this additional exposure to value, size and quality. We should expect to see a number much greater than zero for value and size, as well as a positive value for quality. Finally, any time you run a regression, you won’t be able to explain everything with factor exposure: in the table below this is the “Unexplained” category, and we want this number to be as low as possible.

With that mind, here are the numbers Justin got for two DFA indexes tracking Canadian and US stocks using 17 years of data from January 1999 to the end of 2015:

DFA Canadian Vector Equity Index
Market (beta) 0.98
Value 0.30
Size 0.17
Momentum -0.07
Quality 0.05
Unexplained 0.53%

.

DFA US Vector Equity Index
Market (beta) 1.07
Value 0.47
Size 0.29
Momentum -0.07
Quality 0.17
Unexplained 0.14%
Sources: Dimensional Returns 2.0, AQR Data Library
.

What do these numbers tell us? The relatively high numbers we see for value and size suggest the DFA indexes do a good job at “tilting” toward underpriced and small-cap stocks. The indexes got some additional exposure to the quality factor as well (more in the US than in Canada). Some of the performance is unexplained, but not a lot. Overall, when value, small-cap or quality stocks outperform (or lag) you should expect similar results from these indexes.

How do the iShares Multifactor ETFs compare?

Now let’s consider a different multi-factor strategy available to Canadian ETF investors. The iShares Edge MSCI Multifactor Canada (XFC) was launched in September 2015 along with similar funds for US and international stocks. These ETFs track the MSCI Diversified Multiple-Factor Indexes, which focus on value, momentum, size and quality.

During the same 17-year period (from 1999 through 2015) the Canadian version of this index reported an annualized return of 11.86%, compared with just 7.24% for the broad market. That’s some huge outperformance. But we want to know the reason: did the index really get added exposure to the four factors, and if so, did that explain the much higher return?

When Justin analyzed the Canadian data for a blog post he wrote in August, here’s what he found:

MSCI Canada IMI Diversified Multiple-Factor Index
Market (beta) 0.98
Value -0.03
Size 0.21
Momentum 0.07
Quality 0.07
Unexplained 2.41%
Sources: MSCI, AQR Data Library
.

This result is a head-smacker. The index did indeed have added exposure to small cap stocks, but its huge outperformance could not have come from this tilt, because over the 17 years in question small caps actually lagged the broad Canadian market. The added exposure would have hurt, not helped. There was a wee tilt to momentum and quality (0.07 is pretty modest), but the index actually had less exposure to value stocks than a traditional index fund would have had.

Bottom line: this smart beta index may have delivered market-beating returns when it was backtested, but it’s hard to understand why. Indeed, the amount of unexplained performance was huge at 2.41% per year. Any time you can’t explain the reasons a strategy outperformed in the past, it’s hard to have any confidence it will work in the future.

Other posts in this series:

Smart Beta ETFs: Your Complete Guide

A Brief History of Smart Beta

Understanding the Value Factor

Understanding the Size Factor

Understanding the Momentum Factor

Understanding the Low Volatility Factor

Understanding the Quality Factor

Exploring Multi-Factor Models