Can a Language Model Keep a Tax Lot Straight? A Rules-Grounded Benchmark for Wash-Sale, Holding-Period and Lot-Relief Decisions

Summary

The question we asked

A tax-aware separately managed account makes the same few decisions hundreds of times a year. Is this loss sale a wash sale? If it is, which shares carry the disallowed loss forward, and what does their basis become? Is a gain short-term or long-term? Which lots should the next trim sell? None of these is a hard question. Each has one right answer under IRS Publication 550, and a short program can compute it to the cent. These are also the questions that operations staff now type into chat windows, and that assistant products are being built to answer for people who manage other people's money. We wanted to know how often a language model gets them right, and what a wrong answer costs.

How we built the test

We wrote a small rules program whose every rule is read off Publication 550 (2025), and it reproduces every worked example in the publication that it relies on. That program is the answer key for a 193-item benchmark in five families: whether a loss sale is a wash sale and how much of the loss survives, the basis of a replacement lot, whether a sale is short- or long-term, the gain under first-in first-out or specific identification, and a year-end reconciliation of one ticker across a household's accounts. Table 1 lists them.

Most of the items are not invented. A simulated household trades through 2025 on simulated prices. It holds a direct-indexing account, a second taxable account, a traditional IRA, a Roth IRA, a spouse's taxable account and an adult child's account. Of the items, 133 are random samples of decisions the direct-indexing account faced during that year. The other 60 sit on the lines the rules draw: day 30 against day 31, the anniversary against the day after it, a leap day inside the year, a purchase in an IRA, a replacement split across two accounts.

Each question goes to each model in four settings. Closed-book, the model sees only the household facts, the trades and the question. Rule-grounded, it also sees the governing passages of Publication 550. Program-aided, it writes a short Python program that we run. Delegated, it only transcribes the trades into a fixed schema and the program answers. Two open-weight models running on a laptop, Qwen2.5 7B and Gemma 3 12B, answered all 193 items in every setting. A larger local model, Qwen2.5 14B, and one hosted model, Claude Sonnet 5, answered a 35-item stratified subset. Table 2 lists them.

What came back

Figure 1 shows accuracy by setting. Unaided, the two small local models got 13% and 23% of the items right. Quoting the rules in front of them helped little: both stood at 20%. Writing a program took them to 24% and 26%. The hosted model did much better, 31 of 35 closed-book and 34 of 35 with the rules, and its misses were the kind that do not announce themselves: a wash sale nobody noticed, a purchase on day 31 counted as inside the window.

The delegated setting looked different. Asked only to transcribe the trades and leave the decision to the program, the small models reached 98% and 100%, and the hosted model answered 35 of 35. What the small models still missed came from transcription, usually a mislabelled account.

We priced the wrong answers on the simulated household by weighting each family's mean error by the number of such decisions the account faced in the year. Figure 5 shows the result. An unaided small model would misstate the year's tax by $93,847 to $101,592, about three times the tax value of the year's harvest. Delegation brings that to $0 on the natural items. Table 6 gives the breakdown by family, and the year-end reconciliation carries the largest share of it.

Could constraints replace the program?

Version 2 of the paper takes up an objection to that conclusion. If the rules are hard constraints, why not let the model answer and reject anything that breaks them? Some accounting identities hold for every correct answer and need no tax rule to test. We applied them to the answers we already had. They rejected 72% of the small models' closed-book errors and one of the hosted model's eleven. Figure 6 shows the split. The rejected answers hold most of the dollars. Moving a rejected answer to the nearest admissible one seldom makes it right, though, and Table 7 gives those counts. The checks make a useful screen in front of the program, and no substitute for it.

We also attributed the wrong answers to named misconceptions where one reproduces the answer given, such as ignoring the wash-sale rule altogether or getting the term wrong. Table 5 reports the counts. Most errors matched none of the named classes.

Why the test is built around a household

The wash sales that cost money are rarely inside the one account a custodian's software watches. That software sees a single account. It cannot see a spouse's account at another broker, or an IRA that bought the same stock a week after a harvest. Cross-account facts like these are what an assistant would be asked to reason about, so we built the benchmark around a household rather than one account.

Our practical reading is narrow. A model can read documents and turn a paragraph of trades into rows. The program should make the call, and someone should look over what the model transcribed before it does.

Who this is for: CPAs and tax professionals who see language-model output on wash-sale and lot questions, and journalists covering AI in investing and tax preparation.