A Commodity Index With a Broken Download Link

One data source vanished. That was the whole problem behind a short July 2013 update to a post from the year before: the website that had supplied Thomson Reuters/Jefferies CRB Index history no longer worked, and a backtest that depended on it suddenly had nothing to stand on.

It sounds like a small housekeeping note. We think it points at something every systematic investor runs into sooner or later. Research you build is only as durable as the free data feeding it, and free data has a way of disappearing without warning.

Originally the exercise was to extend commodity time series backward. Two big commodity ETFs, DBC (PowerShares DB Commodity Index Tracking Fund) and GSG (iShares S&P GSCI Commodity-Indexed Trust), only date from the mid-2000s. That gives you one commodity supercycle, one financial crisis and a handful of recoveries. Not much to test a strategy against.

So the trick is to splice. You take the ETF history where it exists, then bolt on the older history of an index that tracks closely, and you end up with a series reaching back decades. That 2013 update was about finding replacement sources for the older piece after the first one went dark.

Why does this matter beyond commodities? Because the same patching problem hits anyone testing a strategy on instruments with short histories. Most ETFs are young. Yet the strategies we want to test them in are old. Bridging that gap carefully is a skill, and the way it goes wrong is instructive.

Two Indexes That Look Alike and Are Not

That update identified two candidate replacements. One was the Thomson Reuters/Jefferies CRB Index, Total Return version, downloadable as a spreadsheet from a chart page. A second was the TR/J CRB Global Commodity Equity Index, also Total Return, downloadable with an "All Dates" option.

Noticing the two series differed, the author admitted not looking hard for the reason. Of the two, the first fit DBC and GSG better. That was the extent of the diagnosis.

A reader supplied the answer in the comments, and it is a good one. That equity index, they pointed out, is an index of listed equities: shares of companies whose business is in commodities. Think producers, miners, energy firms. It is not a basket of futures contracts at all.

That explains why one line sat apart from the others on the chart and, from the commenter's memory, tracked the S&P 500 more closely than it tracked oil or wheat. A company that drills for oil owns a claim on oil, but it also carries debt, management decisions, tax bills and a share price that swings with the general mood of the stock market.

This is the most common trap in splicing. Two things share a name, or a theme, and seem to point the same way. Charts even look roughly alike over short windows. But the return drivers differ, and the difference shows up right when you need it, in a crash.

In 2008, for instance, commodity futures and commodity stocks both fell hard, but equity-linked exposures dropped alongside everything else in the market, while the futures indexes had their own dynamics tied to contango, inventories and demand. Treating them as interchangeable would have given a false picture of how a commodity sleeve behaves in a portfolio.

What a Total Return Futures Index Is Really Measuring

To see why the CRB futures index is the better stand-in for DBC and GSG, it helps to know what these products earn. A commodity futures index does not hold barrels of oil. It holds futures contracts, and it rolls them forward as they approach expiry.

The total return version adds the interest earned on the collateral posted against those contracts. That is a different thing from the spot price. The return has three parts: the change in spot prices, the roll yield (positive when the futures curve slopes down, called backwardation, and negative when it slopes up, called contango), and the collateral interest.

Over long periods the roll component can dominate. A fund sitting in persistent contango bleeds a few percent a year even if spot prices go nowhere. A fund in backwardation collects a few percent a year for the same flat spot price. Neither shows up if you only look at a commodity price chart.

That is why a futures-based total return index matches DBC and GSG better than a stock index or a spot index would. All three share the same machinery: contracts, rolls, collateral. The equity index shares only a theme.

The two ETFs themselves differ too. GSG follows the S&P GSCI, which weights by world production and so leans heavily toward energy, often well over half the index. DBC tracks a smaller basket of fourteen commodities with a more balanced split and a roll rule that tries to pick contracts with better curve shapes. So even among the funds, "commodities" is a loose label, and any spliced history inherits the weights of whichever index you borrow.

The 2013 post did the sensible check: scale every series to start at one, align them on common dates, and plot them together. It is a crude test, but it is the right first one. If the lines diverge in the overlap period, they will not behave in the stretch you cannot observe.

Where Commodity Splicing Touches a Bank-Heavy Portfolio

Now to the part of this that matters for money. Suppose a portfolio holds financial stocks such as MS, BAC, GS and C, plus a technology name like META. The question a longer commodity history lets you ask is how a commodity sleeve would have behaved next to those holdings across more than one market regime.

Take the post-2008 world as the backdrop. Banks like Bank of America and Citigroup lost most of their market value between 2007 and early 2009, with both trading down more than 90 percent from their peaks at the worst point. Goldman Sachs and Morgan Stanley fell less but still suffered deep drawdowns. A commodity futures index also fell sharply in the second half of 2008, so the diversification benefit in that window was weaker than many expected.

But zoom out. In the 1970s, commodities rose while equities stagnated. In the 1980s and 1990s commodities mostly drifted lower while financials boomed. A longer spliced series lets you see those alternating decades. The short ETF record, starting in 2006, cannot show it.

The risks of leaning on a spliced series are real. If the older index has different weights, a different roll methodology or a different treatment of collateral interest, then the back-history is a rough sketch rather than a measurement. Backtests built on it can look cleaner than reality. Also, the 1970s inflation episode may not repeat in the same form.

The opportunity is that a commodity sleeve may offset inflation shocks that hurt both bank balance sheets and long-duration growth stocks like META. Consider a rough scenario. A portfolio holds 40 percent in those five stocks and 60 percent in other assets. Carving out 10 percent for a commodity futures fund would, in a year where energy prices surge and equities fall 15 percent, cushion the drawdown by perhaps one to two percentage points, depending on the correlation you assume.

Change the assumption and the number changes. If commodity returns correlate at 0.6 with equities, as they briefly did in 2008, the cushion nearly vanishes. If correlation is near zero, as in several earlier decades, the cushion is meaningful. Knowing which regime your data covers is the point of extending the history.

Practical Steps for Building and Checking a Spliced Series

The workflow in the original post is straightforward and worth copying. The author used R with the Systematic Investor Toolbox and the quantmod package. Load GSG and DBC from Yahoo, adjust the prices for dividends and splits, then read the index files from disk and convert each into a time series object.

Align everything on the dates where all series exist, rebase to one, and plot. That is the sanity check. The code is about thirty lines, and it will tell you quickly whether a candidate index is a plausible proxy.

Beyond a visual check, we would add a few numbers. Compute the correlation of daily or weekly returns between the index and each ETF over the overlap. Compare annualized return and volatility. Look at tracking error, the standard deviation of the return difference. A high correlation with a persistent return gap tells you the proxy captures direction but misses something systematic, like fees or roll methodology.

Fees deserve attention. DBC and GSG charge expense ratios of roughly 0.75 to 0.85 percent a year, while an index has no costs. Over a long splice, subtracting that drag from the index portion keeps the combined series honest. Otherwise the older history looks better than anything an investor could have earned.

Data hygiene matters as much as method. The source pages in the 2013 post required manual steps: pick the right series, click through to a chart, find the download link. When a site changes, the workflow breaks. Save the raw files locally with the download date and the exact series name, so the research can be repeated years later even if the source is gone.

That is what bit the author here. Having local copies of the spreadsheets would have kept the old analysis reproducible. Notes about where each file came from, and what options were ticked when it was downloaded, would have made the replacement search much faster.

Finally, test more than one proxy when you can. Here the two candidates disagreed, and the disagreement was informative. If a second index disagrees with the first, that is a signal to find out why before building on either.

Taking the Lesson Back to Your Own Research

The surface of this update is a data-source swap. Underneath is a habit: check what a series contains before you let it stand in for another. A name, a theme or a visual resemblance is not enough.

For anyone building or reading backtests, a few steps follow. Keep local, dated copies of every raw data file you use. Record the exact selections made at download. When you splice two series, compare returns and volatility over the overlap, not only the chart, and subtract known fees from the index portion.

Ask what the proxy owns. A futures index and an equity index of commodity producers sound close and behave differently, as the reader comment pointed out. If the proxy is built on a different asset type, expect different crash behavior and different correlation to stocks like MS, BAC, GS, C and META.

When you assess diversification, test it in several regimes. The 2008 window shows commodities and financials falling together. Earlier decades show them diverging. A conclusion drawn from only one of those periods will mislead you.

And keep the uncertainty in view. The original post ended with an open invitation for insight about the gap between the two indexes, and a stranger supplied a clear answer within days. That is a decent model for how to do this work: state what you do not know, show the data, and let others check it.