Two Documents, Same Subject, Wildly Different Lengths

Take a 50-word snippet about MS Dhoni and the full 5,000-word Wikipedia page we clipped it from. Count the shared words and the long page wins against almost anything you compare it with, because it has more words to share. Yet the snippet and the page are obviously about the same thing.

That small puzzle is the starting point of a tutorial on cosine similarity, a metric that measures how alike two documents are regardless of their size. The author wrote the tutorial for text analysis, but we found it speaks directly to a problem investors face constantly: how do you compare two things when one is bigger than the other?

Think about the comparison a portfolio manager makes every quarter. A $40 billion fund and a $400 million fund both hold bank stocks. Looking at raw dollar overlap tells you the large fund holds more of everything, which says nothing about whether the two funds think alike.

Cosine similarity strips out scale and asks only about direction. Two holdings vectors that point the same way score near 1, even if one is a hundred times longer than the other. Two vectors at right angles score 0. That single idea, comparing orientation instead of magnitude, is what we want to explore here, and then apply to a basket of familiar tickers: DIA, EFA, C, GS and BAC.

How the Angle Replaces the Word Count

The mechanics are short. Each document becomes a list of word counts, one slot per word in the vocabulary. In the tutorial's cricket example there are only three slots: how often "Sachin", "Dhoni" and "Cricket" appear. Every document is then a point in three-dimensional space.

The long Dhoni page might be (2, 50, 40) and the snippet (0, 10, 8). Those points sit far apart in straight-line distance. But draw an arrow from the origin to each and the arrows lie almost on top of each other. The cosine of the angle between them is the similarity score, and here it lands close to 1.

The formula divides the dot product of the two vectors by the product of their lengths. Dividing by length is the step that removes size from the comparison. A document that repeats itself ten times over gets longer, but its direction stays put.

A common misunderstanding is that cosine similarity is a better version of Euclidean distance. It answers a different question. Euclidean distance tells you how far apart two things are in absolute terms, which matters if magnitude carries meaning. If one document says "cricket" 50 times and another says it 10 times, and that frequency gap is the thing you care about, distance is the right tool.

Another pitfall: counts of raw words reward common filler. The tutorial points out that a TF-IDF weighting would have been a better choice than plain counts, because it shrinks the influence of words that show up in every document. Without it, words like "the" can dominate the direction of every vector, and everything starts to look alike.

What the Presidents Example Teaches About Meaning

The tutorial's second example uses three short passages about Trump, Putin and a presidential election. Document A covers Trump winning an election and his friendship with Putin. Document B covers Trump defending Putin over election interference. Document C is about Putin's career in Russia.

The expectation is that A and B should be closer than C and B, since A and B share more subject matter. Build a document-term matrix with scikit-learn's CountVectorizer, pass it to cosine_similarity, and that is what comes out. The method recovers a judgment a human reader makes instantly.

Then the tutorial raises a harder case. Add documents about food and the method still works, but it has a blind spot. "President" and "Prime Minister" are different strings, so plain cosine similarity treats them as unrelated. So are "food" and "dish", or "hi" and "hello".

Soft cosine similarity fixes this by bringing in word embeddings, here a FastText model loaded through gensim. Words with similar meanings get a nonzero similarity to each other, and the score now credits a document that says "dish" when the other says "food". The result is a matrix where documents on the same topic show clearly higher scores than documents on different topics.

This distinction is worth carrying over to finance. Exact-match overlap is the hard version: two funds either hold the same ticker or they do not. The soft version asks whether two holdings play a similar role, even if the names differ. A bank stock and a bank ETF sit on different line items, yet they respond to many of the same forces.

Treating Portfolios as Documents

Here is the translation. A portfolio is a document, each security is a word, and the weight in the portfolio is the word count. Two portfolios become two vectors, and cosine similarity tells you how aligned they are, whatever their dollar size.

Consider a small illustration using the tickers in this post. The SPDR Dow Jones Industrial Average ETF (DIA) holds 30 large US companies, and Goldman Sachs (GS) is one of them, carrying a large weight because the Dow weights stocks by price and GS trades at a high share price. Citigroup (C) and Bank of America (BAC) are not in the Dow at all. The iShares MSCI EAFE ETF (EFA) holds developed-market stocks outside North America, so it overlaps with none of these five at the ticker level.

Build vectors over those five names with made-up but plausible weights. A "US financials" sleeve might be C 30%, GS 40%, BAC 30%. A DIA-heavy sleeve might be DIA 100%. A global sleeve might be EFA 100%. Cosine similarity between any two of these sleeves is zero, because they share no coordinates. The vectors are orthogonal, and in the ticker-level view that is correct.

That result is also misleading, and it shows the limit of hard cosine similarity. GS sits inside DIA's own holdings, so the US financials sleeve and the DIA sleeve are far from unrelated. They share a real exposure that the ticker-level vectors cannot see. Looking through the ETF to its constituents, DIA gives GS roughly 7 to 8 percent of its weight in recent years, and the bank sleeve's cosine similarity with DIA rises above zero once we express both in underlying stocks.

So the first practical lesson is about the vocabulary. Compare portfolios at the level of underlying holdings, not wrappers, or the metric will report independence where there is overlap.

Where Overlap Hides and Where Diversification Is Real

Now the soft version. Suppose we replace tickers with factor exposures: sensitivity to interest rates, credit spreads, equity beta, and so on. C, GS and BAC all load heavily on the same few factors. Their vectors point in nearly the same direction, with cosine similarities that would plausibly sit above 0.9 on any reasonable factor set. They are different words that mean nearly the same thing.

That is the portfolio equivalent of "president" and "prime minister". Holding all three banks feels like holding three positions, and by name it is. By exposure it is closer to one position held three times. During the 2008 financial crisis, all three fell more than 80 percent from their peaks at some point, and they fell together.

EFA tells a more mixed story. Its holdings include European and Japanese banks, so some of its direction overlaps with the US bank trio. But it also holds consumer, health care and industrial names that point elsewhere, and it carries currency exposure to the euro, yen and pound. A realistic factor-based cosine between EFA and a US bank sleeve might land somewhere around 0.4 to 0.6. That is real diversification, though far from complete.

Risks in this approach deserve a plain statement. Cosine similarity says nothing about magnitude, so two portfolios with similar direction can have very different volatility. A 3x leveraged version of the same holdings scores a perfect 1.0 against the original. Similarity of direction also changes over time: correlations between stocks rise sharply in selloffs, and a measure calibrated in a calm year can understate overlap in a bad one.

The opportunity is that the metric is cheap, fast and scale-free. You can compare a $5,000 account to a $5 million one, a mutual fund to a model portfolio, or your own holdings to a benchmark, using the same number. That makes it a handy first screen for hidden duplication.

Running the Check Yourself

The workflow mirrors the tutorial. Collect your holdings, look through any funds to their underlying positions, and put weights into a table with one column per security and one row per portfolio. Feed that table to scikit-learn's cosine_similarity function. The tutorial notes it accepts either a pandas DataFrame or a sparse matrix, so the same code handles three holdings or three thousand.

Weighting choices matter. Raw dollar weights let a few large positions dominate the direction, much as common words dominate raw text counts. Some analysts prefer to transform weights first, for example by working with risk contributions rather than dollar amounts, so that they do not ignore a small but volatile position. This is the portfolio version of the TF-IDF adjustment the tutorial recommends.

Timing is worth thinking about. Run the comparison when you add a new fund, when you rebalance, and after any large market move. Weights drift: a portfolio that began as 60 percent equities and 40 percent bonds can sit at 70/30 after a strong year, and its direction in factor space has changed with it. A quarterly check is a reasonable cadence for most investors, and a check after a drawdown tells you how much your holdings really moved together.

Implementation has its own snags. Mutual funds report their holdings with a lag, often 30 to 60 days. We have to estimate factor exposures, usually from a regression over two to five years of returns, and the estimates carry error. Treat a similarity of 0.82 versus 0.85 as the same answer; treat 0.9 versus 0.3 as a real difference.

Decide in advance what score triggers action. One simple rule: if two positions you think of as separate bets score above 0.85 on a factor basis, ask whether you want to pay for both. If a new fund you are considering scores above 0.9 against something you already own, it adds little beyond cost and complexity.

What to Do With a Number Between Zero and One

Cosine similarity belongs in the toolkit because it answers a narrow question cleanly: do these two things point the same way, whatever their size? The tutorial demonstrates it on cricket and presidents, but the idea carries over to any comparison where scale gets in the way, and portfolios are a good example.

The soft version teaches the second lesson. Exact matches miss relatives, and in markets the relatives are everywhere: C, GS and BAC share a direction even though their tickers differ. A metric that only sees identical labels will treat a basket of three banks as diversified, which history does not support.

We suggest a few concrete steps. First, list your holdings and look through every fund to its top 20 or 30 underlying positions. Second, build a simple table of weights and compute pairwise cosine similarities, first on holdings and then, if you can, on factor exposures. Third, flag any pair above roughly 0.85 and decide whether you intended the overlap.

Fourth, remember what the number leaves out. It cannot see leverage, it cannot see volatility, and it will move when markets come under stress. Pair it with a measure of risk contribution before making a change, and rerun it after meaningful market moves.

Used this way, the score is a quick way to catch duplication before it shows up as a loss. It will not tell you what to buy, but it will tell you whether the thing you are about to buy is something you already own under another name.