Probability and statistics for investors

Level 1 Plain words

For every reader: the idea in plain words, no formulas.

Outcomes and chances

An outcome is something that can happen: the market rises 12 % this year, a bond defaults, a company beats its earnings forecast. A probability is how likely it is, from zero (never) to one (certain). When we list every outcome with its probability, we have a distribution. Most of finance comes down to describing distributions well and not mistaking one for another.

What "on average" hides

An average is one number standing in for many. It can mislead. Picture a coin game where heads adds 50 % to your money and tails takes away 40 %. On average each toss gains 5 %, so the game sounds worth playing. Yet a player who gets the usual mix of heads and tails loses money, because a 40 % loss does more damage than a 50 % gain repairs. Level 3 works out how much. The lesson for investors: ask about the spread of outcomes and the typical path, beyond the average.

Why mixing investments helps

Two investments that do not always move together smooth each other out. When one has a bad year, the other may not. The less they move together, the smoother the mix. That idea, measured by correlation, explains why diversification works and why it sometimes stops working in a crisis.

Level 2 How it works

For readers who want the mechanism, with the diagrams.

Beginner path stops here

Conditional probability and independence

A conditional probability is the chance of one thing given that another has happened: the chance of a loss this year given that a recession has started, say. Two events are independent when knowing one does not change the chance of the other. Coin tosses are independent. Market years are close to independent in their averages, though not in their swings, since a volatile month tends to follow a volatile month. Much of what goes wrong in risk models comes from treating linked events as independent, as Level 4 shows with LTCM's trades in 1998.

Distributions investors meet

Four shapes cover most of this series. A binomial distribution counts successes in a fixed number of tries, such as heads in ten tosses or defaults in a pool of loans. A normal distribution describes sums of many small independent effects. A lognormal distribution describes prices that grow by percentages and cannot fall below zero; most option models use it. A fat-tailed distribution, of which there are many kinds, puts more weight on extremes than the normal does and fits daily market returns better. Choosing the wrong shape often hurts a model more than choosing the wrong inputs.

Expected value

The expected value of a bet multiplies each outcome by its probability and adds the results. For a fair six-sided die the outcomes are one to six, each with probability one sixth, so the expected value is 3.5. Nobody ever rolls 3.5. The expected value describes the long-run average over many rolls, which is why it suits a pension fund with thousands of positions better than a person with one house.

Variance and standard deviation

Variance measures spread: the probability-weighted average of the squared distance from the expected value. Its square root, the standard deviation, comes back in the same units as the outcome. For the die the variance is 2.92 and the standard deviation 1.71. In investing we measure returns in percent, so the standard deviation of a stock fund comes out in percent too, near 20 % a year for US large companies over the long run.

The normal curve and the 68 and 95 rule

Many quantities cluster around an average with fewer and fewer cases further out, in a bell shape called the normal distribution. Two numbers describe it fully: the mean and the standard deviation. About 68.3 % of outcomes land within one standard deviation of the mean, 95.4 % within two and 99.7 % within three. For a stock fund with a 10 % mean and a 20 % standard deviation, that says roughly two years in three fall between −10 % and +30 %, and about one in twenty falls outside −30 % to +50 %.

Fat tails and skew

Real market returns depart from the bell in two ways. They have fat tails: very large moves, up and down, happen far more often than the normal curve allows. They also have skew: for stock indexes the big moves tend to be falls, with crashes sharper than rallies. Daily returns show both more strongly than yearly ones. A model that assumes normal returns will understate the chance of a disaster, sometimes by a factor of thousands.

Correlation

Correlation measures how two returns move together, on a scale from −1 to +1. At +1 they move in lockstep; at 0 knowing one tells you nothing about the other; at −1 one rises when the other falls. Stocks of companies in the same industry tend to show high correlation. High-quality government bonds and stocks have often shown low or negative correlation, though not always, and 2022 was an exception, when both fell together.

Diversification is a correlation story

Mixing two assets lowers risk only when their correlation sits below one. At a correlation of one, a 60/40 mix of stocks and bonds is as risky as the weighted average of the two. As the correlation falls, the mix gets less risky than that average, and at negative correlations much less. The gain from diversification lives entirely in that gap.

Sampling error

Even forty years of returns form a small sample. If yearly returns swing by 20 %, the average of ten of them is itself uncertain by several percentage points. That uncertainty, the standard error, shrinks only with the square root of the number of years: four times the data halves it. Anyone quoting a ten-year average as if it were the true long-run return is claiming more than the data supports.

Level 3 The math

For Series 65 and CFA readers: formulas and worked examples.

Series 65 path stops here

Expected value and variance

For outcomes $x_i$ with probabilities $p_i$ that add to one, we define the two quantities as follows.

In words, the expected value is the sum of each outcome multiplied by its probability.

In words, the variance is the probability-weighted average of each outcome's squared distance from the expected value.

Arithmetic and geometric returns

The coin game shows a general rule. When yearly returns have mean $\mu$ and standard deviation $\sigma$, the compound growth rate falls short of the mean by about half the variance.

In words, the geometric mean is roughly the arithmetic mean minus half the variance.

For a stock fund with $\mu = 10\%$ and $\sigma = 20\%$, the compound rate comes to about 8.0 %. The approximation holds well for moderate volatility. It comes from treating log returns as normal, the lognormal distribution model of prices that the Black–Scholes option formula also uses.

Two-asset portfolio risk

For weights $w_1$ and $w_2$ that add to one, standard deviations $\sigma_1$ and $\sigma_2$ and correlation $\rho$, the portfolio variance has three terms.

In words, portfolio variance is each asset's weighted variance plus twice the weighted covariance between them.

With many assets the same logic gives a floor. An equal-weight portfolio of $n$ bets, each with standard deviation $\sigma$ and the same pairwise correlation $\rho$, has this risk.

In words, as the number of bets grows the first term fades and the risk settles at the standard deviation times the square root of the correlation.

Twenty bets of 10 % risk each give a portfolio risk of 2.2 % when they are independent, 4.9 % at a correlation of 0.2 and 7.2 % at 0.5. Even a thousand bets at a correlation of 0.5 leave 7.1 %. Adding positions cannot diversify away a shared correlation.

The chance of a loss over a horizon

If yearly returns are independent and normal with mean $\mu$ and standard deviation $\sigma$, the average over $n$ years has standard deviation $\sigma / \sqrt{n}$. The chance that the average comes out below zero is then a normal tail probability, where $N$ is the normal cumulative distribution.

In words, the chance of a negative average return over n years is the normal probability of landing more than mu times root n over sigma below the centre.

The standard error of an average

The standard error of a sample mean of $n$ independent years is the yearly standard deviation over the square root of $n$.

In words, the standard error is the yearly standard deviation divided by the square root of the number of years.

With $\sigma = 20\%$ and ten years, the standard error is 6.3 percentage points. A two-standard-error band around a measured 10 % average runs from −2.6 % to 22.6 %, so ten years of data cannot tell a great asset from a poor one. With fifty years the lower edge rises to 4.3 %.

Counting successes with the binomial

The chance of $k$ successes in $n$ independent tries, each with probability $p$, is the binomial formula.

In words, the chance of k successes is the number of ways to choose which tries succeed, times the chance of k successes and n minus k failures in one such order.

For ten fair tosses the chance of five heads is 24.6 %, and the chance of seven or more heads is 17.2 %. Apply that to fund managers: among managers with no skill and an even chance each year, about 17.2 % will beat their benchmark in seven or more of ten years by luck alone. A good ten-year run proves less than it seems.

Value at risk and expected shortfall

Value at risk answers a narrow question: how much could a portfolio lose over a period at a given confidence level? Under a normal model it is a fixed number of standard deviations below the mean, with $z = \text{−1.645}$ at 95 % confidence.

In words, value at risk is the loss at the point where only one minus c of outcomes are worse, scaled by the portfolio's value.

Expected shortfall goes one step further and asks for the average loss in the bad cases beyond that point. Regulators moved bank trading rules toward it after 2008 because value at risk says nothing about how bad the tail gets.

Bayes' rule and a recession signal

Bayes' rule updates a probability when new evidence arrives. For an event $R$ and a signal $S$ it reads as follows.

In words, the chance of the event given the signal equals the chance of a true alarm divided by the chance of any alarm, true or false.

Level 4 History, analysed

For readers who want past episodes run through the Level 3 math.

19 October 1987

We can measure how badly the normal curve failed. A fall of 20.5 % against a daily standard deviation near 1 % is a move of 20.5 standard deviations. On a normal curve, a day that bad has a probability of about ten to the power −93. Since the universe has existed for fewer than ten to the power 18 seconds, that model says the crash should never have happened. Even a five-standard-deviation down day should come about once in 13,843 years of trading under the normal curve, yet the S&P 500 had several daily moves of 5 % or more in October 2008 and again in March 2020. So the data rejects the normal model for daily tails. What it does not reject is the 68 and 95 rule as a rough guide for ordinary years, which is how the overview article uses it.

A second lesson from 1987: a model can feed a fall. Portfolio insurance, a strategy that sold index futures as prices fell, was widely used in 1987, and the selling it called for added to the pressure that day. Chapter 16 returns to that link.

1998: correlations break

LTCM held hundreds of positions, many of them bets that two similar bonds would converge in price. In normal markets those bets were close to independent, so the fund's total risk looked small. The many-bets formula shows why that view was fragile. With twenty bets of 10 % risk each, independence gives a portfolio risk of 2.2 %. If a panic pushes the correlation to 0.5, the same positions carry 7.2 %, more than three times as much, with no trade made. Heavy borrowing then turns that jump into a threat to the fund's survival. We take the full case, with its leverage, in Chapter 15.

2008: correlations go to one

Take a US investor with half in US stocks and half in foreign stocks, each with a 20 % standard deviation. At a correlation of 0.5 the mix has a standard deviation of 17.3 %. If the correlation rises to 0.9, the figure climbs to 19.5 %, close to holding a single market. Correlations measured in calm periods tend to understate those in crises, which is when diversification is wanted most. High-quality government bonds did better that year: they rose while stocks fell, which is the reason bonds sit in most portfolios. Our view: diversify across kinds of risk, such as interest rates and inflation as well as company profits, because diversifying across countries alone gives less protection in a global panic.

What the three cases have in common

Each episode broke an assumption from Level 3. The 1987 crash broke the normal tail: a move the model put beyond any lifetime happened in one afternoon. LTCM's bets broke independence, since trades that looked unrelated shared a common driver, the willingness of traders to hold risk at all. The autumn of 2008 broke stable correlation, as the diversification measured in calm years shrank when it was needed. A risk model built with care can still go wrong in all three ways at once, and stress tests that set correlations near one and returns several standard deviations down are the usual answer. Such a test asks what the portfolio loses if the bad case arrives, without asking how likely it is. For a household the same idea applies on a smaller scale: check that the plan survives a 2008-sized fall in the year before retirement.