Why One Transformer for Pictures and Words Is a Hard Sell

Most multimodal models are assembled from parts. A text tokenizer feeds a language model, a separate image encoder feeds the same model, and engineers stitch the pieces together afterward. The JetFormer paper (arXiv 2411.19722) asks whether that stitching has to happen at all.

Its answer is a single autoregressive transformer trained on raw images and text together. A normalizing flow handles the image side, and the whole system trains as one. The authors suspect separate encoders may be a weakness, since each one serves its own job and can discard information the other modality would have wanted.

We find that argument plausible, though not settled. Pretrained encoders are popular because they are cheap, well understood and good. Replacing them with something trained from scratch is a bet, and the interesting question is what the bet buys.

The paper also continues a thread from earlier work on GIVT, which used a variational autoencoder to produce continuous image tokens. JetFormer drops the autoencoder and uses a flow instead. That swap sounds small but it changes what the image representation can lose.

Why should readers of a finance blog care? Banks, insurers and asset managers sit on enormous piles of documents where text and images are tangled together: scanned forms, check images, charts pasted into research notes, tables embedded in filings. A model that reads and writes both modalities natively is the kind of tool those institutions will eventually want. We think it is worth understanding how it works before the vendor pitches arrive.

What a Normalizing Flow Does to an Image Patch

A normalizing flow learns a mapping from data to a simple distribution, say a standard Gaussian. The mapping is invertible, so it throws nothing away: you can run it forward to get a compact representation, and backward to recover the original image with no error. That property is called lossless, and it is the main reason the authors chose this tool.

In practice we build the mapping by composing many small, simple transformations, each acting on a subset of the dimensions. We train every one of those steps jointly, with the goal of maximizing the likelihood of the data under the combined mapping. Stacking them gives the flow enough flexibility to model something as complicated as a photograph.

JetFormer applies this idea patch by patch. Each patch of the image goes through the flow, and the output is what the paper calls a visual token. Text tokens are the ordinary kind we all know from language models.

Here is a way to picture it. Take a 256 by 256 image cut into 16 by 16 patches, which gives 256 patches. Each one becomes a continuous vector, and the transformer reads those 256 vectors the way it would read 256 words.

A common misconception is that a lossless mapping must be wasteful, as if keeping everything means keeping every pixel's noise. The training objective pushes against that, because it rewards the flow for producing representations the transformer can predict well. Whatever is hard to predict costs likelihood, so the system has a built-in reason to organize the information sensibly.

Order of Input and the Gaussian Mixture Output

One detail in the paper deserves attention: the order in which we feed data to the transformer matters. When the text description comes first and the image follows, we train the transformer to produce the visual tokens, which means it generates an image conditioned on a caption. When the image comes first and the description follows, we train it to produce the text tokens, which means it captions an image.

So a single set of weights learns both directions of the task. That is a tidy design, and it is the practical payoff of the one-model philosophy.

The output side borrows from GIVT. Text tokens come from a normal softmax over a vocabulary, but visual tokens are continuous vectors, so a softmax does not apply. Instead the model predicts the parameters of a Gaussian mixture for each visual token, and then samples the actual token from that mixture.

Why a mixture rather than a single Gaussian? A single bell curve forces the model to commit to one most-likely value and a spread around it. Real image patches are often ambiguous: the next patch might be sky or a rooftop, and a mixture lets the model keep both options alive with different weights.

The cost is sampling noise. Draw from a mixture and you get variety, which is good for generation, but you also get occasional odd draws. This is a reasonable tradeoff in a research prototype and one that anyone deploying such a model in a regulated setting would have to examine closely.

Starting With Blur: The Noise Curriculum

The third ingredient is a training schedule. The authors add noise to the visual data, and only the visual data as far as we can tell, then reduce that noise as training proceeds. This is a form of curriculum learning.

The logic runs like this. With heavy noise, fine texture is invisible, so the model can only learn the coarse structure of a scene: where the horizon is, which blob is the subject, how light falls. As the noise shrinks, finer details become learnable, and the model builds on the coarse knowledge it already has.

The authors also describe this as a way to improve the robustness of the representations the autoregressive model produces. We take that as a guard against a particular failure. Without it, a model trained on raw pixels can spend its capacity modeling high-frequency detail that carries little meaning, such as sensor grain, and neglect the global structure that people care about.

A human analogy helps. A student learning to draw usually starts with gesture sketches and proportions before shading skin pores. Doing it the other way around produces drawings with beautiful texture on a figure with the wrong shape.

There is a caveat. Curricula add hyperparameters: how much noise to start with, how fast to decay it, when to stop. Each of those is a place where results can shift, and a method that needs careful tuning is harder to reproduce than one that works without extra setup.

What This Means for Citigroup and Bank Portfolios

Let us be direct about the investment link. A research paper on image and text generation does not move the share price of Citigroup (C), and we do not claim it does. The connection is indirect: it concerns how quickly large banks can turn document-heavy operations into automated ones, and which technical approaches might get them there.

Citi is a useful example because of its scale and its sprawling operations across payments, trade finance, custody and compliance. Much of that work involves reading messy documents. If a unified model can read a scanned trade document, extract the fields, flag anomalies and generate a written summary, that touches several cost lines at once.

Consider an illustrative scenario with round numbers. Suppose a large bank carries roughly $50 billion in annual operating expenses, and document-heavy automation trims 1% of that over several years. That is $500 million pre-tax, or around $375 million after a 25% tax rate. Spread over something like 1.9 billion shares, it adds about 20 cents per share to earnings.

Twenty cents is meaningful but not dramatic, and it would arrive slowly. If the efficiency gain were 0.25%, the figure drops to a nickel. The sensitivity is the lesson: the benefit scales with how much of the cost base is automatable, and that fraction is uncertain.

Risks deserve equal space. Generative models can produce confident errors, and a bank that automates a compliance step with a model that occasionally invents a field value faces regulatory and legal exposure. A single enforcement action can cost more than years of savings, so the realistic path is human review layered on top, which erodes the headline gains.

The opportunity side is also plausible. Banks with large proprietary document archives and strong data governance may be better positioned than smaller rivals to train or fine-tune such systems. Scale in data is an advantage if it can be used legally and safely.

How Investors Can Follow This Without Becoming Researchers

Nobody needs to read every arXiv paper. What helps is a short list of questions to ask when a bank, or an analyst covering one, starts talking about multimodal AI.

First, ask whether the claimed savings are tied to a named process and a measured baseline. A statement like "we reduced manual review of trade documents by a given percentage" is checkable, while "we are investing in generative AI" is not. Earnings calls from large banks regularly include some of each, and the first kind is worth far more.

Second, look at the cost of building against the cost of saving. Training and serving large models is expensive, and technology budgets at big banks already run into the billions. If a project costs $200 million to build and saves $150 million a year, payback takes under two years, which is fine. If the saving is $30 million, the same project looks poor.

Third, consider timing. Research results like JetFormer take a long road to production: the architecture must be reproduced, scaled, evaluated for errors, approved by risk teams and integrated with legacy systems. A gap of three to five years between a paper and a measurable earnings effect is typical in our experience, and often longer in banking.

That timeline argues against trading on AI headlines. Positions in C or any other large bank are better judged by capital strength, credit quality, fee income and the progress of restructuring efforts, with AI efficiency as a modest tailwind layered on top. If a portfolio manager is paying for an AI premium in a bank stock, we would want to see what the premium assumes.

For readers who hold C through a broad financial-sector fund, the exposure is already there and little needs to change. Concentrating further because of a research paper would be a leap that the evidence does not support.

Reading the Technical Signal Correctly

Back to the paper itself, because it shows why progress in this field tends to look the way it does. JetFormer's contribution is an architectural simplification: fewer separate components, one training objective, one transformer. Simplification is often how technology gets cheaper, since fewer parts mean fewer things to maintain and fewer places for errors to creep in.

If the approach holds up at scale, the downstream effect would be lower cost per document processed. Lower cost widens the set of tasks where automation pays, which is the mechanism by which a research idea eventually reaches a bank's expense ratio. The chain is long, but each link is understandable.

There are open questions we would want answered. Does the end-to-end flow match the quality of strong pretrained encoders at comparable compute? Do the Gaussian mixture outputs stay reliable on dense, text-heavy images like financial statements, where a single wrong digit matters? The paper's setting is general image and text modeling, and documents with tables and small print are a harder test.

We also note that competing approaches are advancing quickly. Other groups are pursuing discrete image tokens, diffusion-style generation and large vision-language models built from pretrained parts. JetFormer is one bet among several, and it may influence the field even if it does not become the dominant design.

For a reader tracking this space, the sensible stance is curiosity without commitment. Follow the benchmarks, watch for reproduction by independent groups, and notice when a vendor to the banking industry starts citing end-to-end multimodal training as a selling point.

Steps Worth Taking This Quarter

The analysis points to a few concrete actions for investors, none of which involve rushing into a trade.

Check how much of your portfolio sits in large banks, including indirect holdings through index funds and sector ETFs. For many investors, C is already present through broad US equity funds, and the weight may be around a fraction of a percent of the total. Knowing the number keeps any AI-driven enthusiasm in proportion.

Read the technology and efficiency sections of the next Citigroup earnings materials with the questions above in hand. Look for named processes, measured baselines and stated timelines. Write down what management promises now so you can compare it with results two years from now.

Set a rule for yourself on AI claims: treat a research milestone as a reason to learn, and treat a disclosed, audited cost reduction as a reason to adjust a valuation. The two are separated by years and by many failure points, and conflating them is how investors end up overpaying for stories.

Finally, keep a small reading habit. One paper summary a week, even one outside your usual territory, builds enough fluency to judge when a technical claim is solid and when it is marketing. JetFormer is a good place to begin: a clear idea, a stated mechanism, and honest tradeoffs around noise schedules and sampling.

None of this changes a financial plan by itself. What it changes is the quality of questions you ask when someone tells you a bank is about to be transformed by AI, and good questions are cheap to carry and valuable to have.