Ingredients of Index Construction

Indexes aren’t just important for index funds. By creating a virtual portfolio that represents the “market” (or a specific style of sector), they also establish a “typical” return and risk level that active mutual funds can be benchmarked to.  

According to ICI, mutual funds have grown over the past 10 years and account for investor assets totaling around $13 trillion (Chart 1). Their data also shows the growth in index funds, which track the index more closely, rising from 8% of the market to 18% in just the last decade. 

On top of that, pension funds and international investors also care about U.S. equity market performance. 

Because stock prices reflect the future earnings and returns of companies in the market, the return on the index is a useful measure to assess company activity levels and the economy. That’s why business shows on TV tell you what happened to indexes like the Nasdaq-100® and the S&P 500 each day. 

Chart 1: Index and active mutual funds both depend on index construction 

But how are indexes put together? And should companies care if they are in an index or not? 

What’s the difference between an index and an index fund? 

Indexes can be thought of as “virtual portfolios.” You cannot invest directly in an index. 

In contrast, mutual funds (including ETFs) represent actual investments. That also means they need to buy, hold and sell stocks and reinvest dividends to earn returns. Investors in mutual funds also incur the costs of holding investments and trading – from spread and impact costs to settlement and custody costs, as well as portfolio management expenses. 

Because an index does not need to trade and does not include management costs, index returns are “frictionless” and reflect hypothetical performance before costs. 

What’s the difference between index funds, ETFs and mutual funds? 

The large majority of ETFs are mutual funds. 

There is a larger difference in how mutual funds manage portfolios – index (passive) and actively. Both care about the index they are tracking, but for different reasons (see Chart 2): 

• Index (passive) funds are designed to track an index very closely (as we show later in Chart 8). Portfolio managers need to track the returns of the index very closely, which typically requires them to closely copy almost all the changes that the index makes – while also managing liquidity for daily cash flows (such as inflows and outflows from investors) and cash flows from dividends. Passive funds are evaluated based on how well they track their specified benchmark (i.e., low tracking error). Most large index portfolios also use market cap weightings, which reduces the need to trade (as a company’s price increases, so does its weight in the index and the portfolio – requiring no trading). 
• Active funds are expected to beat (or outperform) the index. This is done by allowing the portfolio manager discretion to vary their portfolio holdings from index weights, sometimes even owning stocks that are not in the index, in pursuit of higher returns or alpha. 

Chart 2: Index (the benchmark) vs. index and active mutual funds 

Importantly, not all ETFs track an index. These days, some ETFs are managed just like active funds – while others track indexes that are constructed in ways that look more like active stock selection. 

How are indexes built? 

All indexes begin with a description or Investment Objective, which is meant to define the goal of the index (i.e., what type of exposure the strategy is meant to offer). The index methodology is created to outline the rules used to construct and maintain the index that reflects the investment objective.  

Indexes can be made up of stocks (equities), options, bonds (like the Bloomberg Aggregate Bond Index), commodities and even currencies. There are even indexes on combinations of assets (like the Nasdaq-100 Quarterly Protective Put Index). 

Almost all indexes use a systematic, rules-based approach to their virtual portfolio construction. For example, the methodologies for any of Nasdaq's thousands of indexes can be found through the Nasdaq's Global Index Website. 

The index provider applies those rules to compile the index. There are typically several key elements that go into creating an index (Chart 3): 

• Base Universe: The entire universe of securities used to pick securities for the final portfolio. 
• Eligible Universe: The subset of securities from the starting universe that passes various screens (e.g., minimum size, minimum liquidity, revenue growth, domicile or headquarters country, listing country, type of security, etc.) 
• Weighting Method: How the portfolio will allocate weight to the eligible securities (market-cap, equal weight, factor-based). 
• Rebalance and Reconstitution Dates: When the portfolio will be re-evaluated to ensure it’s still following the rules closely. That ranges from adding or dropping securities from the index (due to no longer meeting eligibility criteria) to deciding if their weights in the portfolio need to change. 

Chart 3: Index creation process 

For example, if you wanted to build a U.S. large-cap index, you would start with all U.S. stocks and pick the biggest ones. You might then “market-cap weight” the index so the size of the company in the portfolio reflects its size in the market (this has the added benefit of reducing portfolio turnover and, therefore, transaction costs).  

You might think that would make all U.S. large-cap portfolios the same, but as we show below, there are a number of differences that make each U.S. large-cap index different. 

What is a “U.S. stock”? 

It might sound simple to work out what stocks belong to a U.S. country index. But the way some companies are set up can make even that process ambiguous – especially with global firms that have listings in multiple countries. 

For example, S&P starts by defining U.S. companies as firms that:  

1. Regularly file statements with the SEC, AND  
2. Derive most fixed assets and revenues from the U.S., AND 
3. Have a primary listing on an eligible U.S. exchange. 

However, factors like where a company’s headquarters, incorporation and officers are can matter too. 

Russell instead starts by defining U.S. companies as firms that have:  

1. Incorporation in the U.S., AND 
2. Headquarters in the U.S., AND 
3. Most liquid trading in the U.S. 

Although, things like where assets and revenues come from can help if not all three above rules are met. 

On the other hand, Nasdaq-100 requires that the company is listed on the Nasdaq Exchange, which can include ADRs and foreign companies (although companies organized outside the U.S. must have listed options in a registered U.S. market). 

Chart 4: How different indexes determine country classification 

In contrast, MSCI is typically considered a more “global” index provider – and it allocates country designations to all stocks around the world based first on where the company is incorporated and where the primary listing is located. 

Size Matters 

One reason for differences is what size represents “large-cap” companies. 

For example, the S&P 500 is, by definition, exactly 500 very large companies. However, they aren’t the 500 largest by rank for a couple of reasons: 

• S&P holds new companies out of their index, as a rule, until they display consistent profitability (S&P requires that the total net income of companies must be greater than 0 over the last four quarters, including the most recent quarter, in order to be eligible for inclusion). This explains the long time to add companies like TSLA and PANW. 
• S&P also tries to minimize turnover for their clients managing actual portfolios. So, they don’t automatically add companies to the S&P 500 as soon as they become “large enough” (which, as of July 2023, was around $14.5 billion). S&P also doesn’t do regular “reconstitutions” to force a re-ranking and inclusion of all stocks. Instead, S&P 500’s adds and deletes are performed on an ‘ad-hoc’ basis, as determined by the Index Committee. 

In contrast, the Russell 1000 does do an annual reconstitution, where they re-rank all companies in the U.S. and put (most of) the Top 1,000 stocks into the Russell 1000 for the next year. However, they, too, have “buffer zones” to minimize turnover, so a company needs to be much larger to be added than the company being deleted. 

You can see the impact of these decisions in Chart 5, where each dot represents a company – and the top row shows all companies (from the Nasdaq U.S. Benchmark™) ranked and sorted (by color) into the top 100, 500 and 1000 names. This shows that the: 

• Nasdaq-100 includes some companies ranked around 500th in the market with a median market cap of around $55 billion and the smallest stock around $12 billion. 
• S&P 500 includes some companies ranked outside the top 1,000 (yellow dots) with a median market cap of around $31 billion. 
• Russell 1000 includes companies ranked well outside the top 1,000 with a median market cap of around $13 billion. 

Chart 5: Each U.S. large-cap index includes a different range of company sizes 

Liquidity matters 

Given indexes are being tracked by trillions of dollars of investment, trading costs and the ability to actually replicate the index with a real fund are important. Because of that, most indexes also screen out illiquid stocks – as those would be harder and more expensive for index (and active) funds to buy.  

However, what is considered “liquid” differs. For example: 

• S&P (based on the S&P Composite 1500) screens stocks based on the ratio of Liquidity-to-Market Cap (called the Float-Adjusted Liquidity Ratio, FALR) of at least 0.75 and six-month ADV of 250,000 shares.  
• Nasdaq-100 requires 200,000 shares per day over the past three calendar months.  
• Russell requires stocks in the base universe to have ADVT greater than the global median. 

We can see how liquidity and market cap screens work to exclude low turnover and small stocks (grey dots) from the Russell 3000 universe (blue dots) below. 

Chart 6: Illiquid stocks are also excluded from many indexes 

How do different indexes compare? 

Every index is unique in its construction. As we have discussed above, even the Nasdaq-100, S&P 500, and Russell 1000, which are generally similar in size, style, and geographic exposure, are different in their composition. 

There are other differences in how indexes are maintained and updated for changes in the market as well. Including:  

• Additions and deletions of stocks range from an annual reconstitution (Nasdaq and Russell) to an ad-hoc basis (S&P). In addition, most indexes use “buffers” around the market cap cut-offs to stop companies near the cut-off from adding and deleting on alternate years just because of small price changes. These make it harder to get into an index – but once you’re in, it’s also harder to fall out – which, in turn, reduces turnover and trading costs. 
• IPO additions: Russell fast-tracks IPO additions each quarter. Other providers require stocks to be “seasoned” to avoid unnecessary turnover. For example, S&P requires IPOs to trade for at least 12 full months, whereas the Nasdaq-100 requires trading for three full calendar months, not including initial listing. MSCI will also add in IPOs between the scheduled quarterly reviews (subject to trading and size criteria). 
• Shares outstanding changes due to secondary issues and repurchases, and free float can change too. A stock’s size in the index is often = shares x float x price, so changes in both shares and float matter to a stock’s index weight. These adjustments are made at different times depending on each provider’s rules. 
• Most indexes adjust for M&A, spinoffs and dividends as they occur. 
• Sector and Style calculations are not always done at the same time – with S&P recalculating style factors in its December quarterly rebalance. In fact, the four popular index providers use two different style classification rules (GICS and ICB). One special rule for the Nasdaq-100 is that it explicitly screens out securities that are classified as Financials according to ICB classifications. 

Chart 7: Key differences in construction and rebalancing of popular indexes 

When do indexes normally ‘change’? 

Updating the holdings and weights of a portfolio is a key part of index construction. As companies change, updating the holdings ensures that the benchmark is still meeting its investment objective. This is especially important for factor-based indexes, where constituents can fall out of favor with a factor at any time between the portfolio's rebalance dates (known as style drift). 

However, as we noted above, an index doesn’t trade.  

Instead, the index just adjusts the stocks in the portfolio, each security’s shares and the factor or style weightings. 

When do index funds normally ‘trade’? 

As we mentioned above, index funds own stocks for investors and need to track the index very closely. 

We can see how closely it is possible to track an index by looking at the QQQ ETF versus the Nasdaq-100 index (Chart 8). This shows the index fund almost perfectly tracks the index over more than a decade.  

The way a portfolio manager typically does this is by owning all stocks in the index at weights almost identical to the index weights. 

That also means that when an index changes, all the index funds need to do a matching trade. 

Chart 8: Index funds track their index very closely 

Typically, changes to an index become effective after the market closes. That means the index will have a refreshed list of constituents and company sizes starting at the next open (known as the “effective” date). As we show in Chart 9: 

• The price an index provider uses to remove and add stocks is the close of the night before the effective date (in the case below, on Friday) 
• If an index is set to add a new company on Monday, the return of that company from the prior close to the open will be included in the index return (purple dashed line). 
• To match this return, the fund needs to sell the delete at the market close of Friday to replicate what the index is doing. 

In fact, we see evidence of index funds trading in the close to match the index returns more precisely, with much of the expected index volume transacting the instant the market closes. That makes index rebalance day closes unusually large liquidity points (also see the spike in Chart 11).  

Chart 9: Hypothetical example of index reconstitution 

However, even the dates that different indexes choose for rebalances differ too. 

Nasdaq and S&P rebalance on the third (Expiry) Friday of the quarter (March, June, September, or December). This is to coincide with higher liquidity in the markets caused by Triple Witching when multiple derivatives products expire simultaneously.  

Although for adds and deletes, S&P usually uses the date of a stock’s removal (often from M&A closing) to make new additions – allowing cashflows and turnover to match around specific events. 

In contrast, other Russell and MSCI indexes intentionally move reconstitutions away from dates where index trading might impact other trading. For example: 

• Russell picks the fourth Friday in June (unless that’s also quarter-end or year-end when volumes are light). 
• MSCI picks a non-quarter month end (February, May, August and November). 

How much notice of the changes do index providers give? 

Because the whole industry tends to do index trades at the exact same time, index trades can be very large. That means the industry often needs time to position liquidity to meet index funds demand – it would be hard to trade a week’s worth of liquidity in just a couple of days without market impact. To accommodate this, typically, indexes will provide advanced notice of changes – ranging from days (for most S&P changes) to over a month (for Russell’s annual reconstitution). 

Even before that date, though, especially for large rebalances, the index provider needs some time to do their calculations. Typically, an index provider will (Chart 10): 

• Snap data at a specific date.  
• Calculate the new index using data from that date. 
• Announce the changes with enough time for the market to provide liquidity before the index trade date. 

Chart 10: Hypothetical index reconstitution cycle 

Companies should want to be in an index (or three) 

Being included in an index is important for companies. 

A large proportion of long-term investors are professionals. Adding a stock to an index typically forces professionals to consider whether they can still afford not to hold the stock. That’s because if the stock outperforms, and they don’t own it, their portfolio will underperform other funds that do hold the stock.  

Not surprisingly, our studies of Russell inclusions show that index inclusion almost always brings new long-term investors into a company's shareholder base and, with that, increased liquidity and trading (Chart 11). Just the expectation of index inclusion seems to increase liquidity as hedge funds prepare for index buying to come (January-April). Then, confirming what we noted above, volumes spike on the actual June rebalance day, as index funds all add the stock to their portfolio when the index changes. However, once index funds are owners of the stock, they shouldn’t need to trade – yet liquidity seems to remain around twice as high after being added to the Russell 2000 index. 

Chart 11: Liquidity increases when stocks are added to an index 

Other work we’ve done shows that more liquidity usually results in tighter spreads, lowering trading costs and ultimately reducing costs of capital. 

Indexes are important to investors, too 

Although not all indexes are built the same, most follow a few key ingredients in their construction. 

They provide a systematic and transparent, rules-based way to measure the performance of a specific market. They help investors measure the performance of both active and index mutual funds – and maybe even their personal portfolios. 

They have also helped spur the growth of low-cost index investing and ETFs.  

In short, indexes play a vital role in the way that people save and the wealth they generate over time.