What transfers
from text to vision?
Capability Scaling Laws and Transfer Dynamics for VLMs
Predict how a language backbone will learn from multimodal data—using its text capability and a planned training budget.
Text capability S
What the language model knows
Multimodal data D
How much it will learn from
A learning curve,
before training.
Across scales. Across families.
Text capability and multimodal performance move together.
Stronger text capability tends to accompany
stronger multimodal performance.
As an initial observation, we examine 17 LLM–VLM pairs from the OpenCompass leaderboard. Language backbones with higher text benchmark scores tend to have VLM counterparts with higher multimodal scores, revealing a positive correlation between the two capabilities.
This observation motivates our central question: can textual capability help us predict multimodal performance? The correlation alone does not explain how much capability transfers or how a backbone learns from additional multimodal data. Our scaling law models these two effects jointly.
Transfer sets the start. Absorption shapes the slope.
One capability score and one data budget predict the multimodal accuracy trajectory.
S · Text capabilityA learned, unit-norm projection of PCA-compressed text benchmark scores.
B̂ = B₀ − BₘS · AbsorptionStronger textual priors reduce the marginal benefit of additional multimodal data.
P₀ · BaselineThe fitted intercept captures benchmark-specific baseline performance.
Measure
Evaluate the backbone on text benchmarks and center its scores.
Project
Compress with PCA (≥95% explained variance), then compute the learned capability score S.
Predict
Keep the fitted projection and coefficients fixed. Supply a data budget to obtain a learning curve.
During fitting, capability weights and scaling coefficients are learned jointly from observed VLM trajectories using Huber loss and a nonnegative-absorption penalty. At inference, only text scores and the planned token budget are needed. (§2, Appendix E)
Choose the evidence.
Inspect the fit.
Filter real checkpoints and see how the scaling coefficients and fitting error change.
Fitted vs. observed accuracy
Hover, focus, or click a checkpoint to inspect its fitted value.
The horizontal axis is ÂS + B̂ ln D, where B̂ = B₀ − BₘS. P₀ is excluded from the horizontal coordinate but remains in the fit; the reference line is y = x + P₀.
 0.2340B₀ 0.0154Bₘ 0.0092P₀ 0.4340How this interactive fit relates to the notebook
Real observations come from the supplied text_accuracy.csv and multimodal_accuracy.csv: 32 backbones × 8 checkpoints, plus two 72B holdouts × 4 checkpoints. Tokens = checkpoint step × 8,192 × 512 / 10⁹.
The PCA mean, 12 components (≥95% variance), and normalized capability direction are recovered from the notebook’s original 32-model training pool and saved weights. They remain fixed during interaction. Your filters refit Â, B₀, Bₘ and P₀ with constrained least squares, enforcing nonnegative Â, Bₘ and absorption across the selected training S range. The notebook’s Huber δ = 1 objective is quadratic for these accuracy residuals, but the browser does not rerun its joint weight learning, regularization, Differential Evolution or L-BFGS-B.
The default all-32 fit is recovered from rounded saved notebook weights, so small differences from the paper are expected. Removing a training model or family here excludes its observations only from the coefficient refit; it is not a new independent leave-one-family-out experiment because S was originally learned from all 32 backbones.
One law. Unseen scales and families.
A controlled training recipe isolates the effect of the language backbone.
A shared predictor
across 32 backbones.
Mean absolute error in multimodal accuracy, without family-specific recalibration.
The fitting pool includes 32 backbones. Qwen2.5-72B Base and Instruct are held out, bringing the full study to 34.
Entirely new families.
Consistent scaling trends.
Fit on the remaining families, then predict all trajectories of the held-out family.
What the coefficients reveal.
Text capability is multidimensional. More of it does not always transfer in the same way.
Some text benchmarks
impose a transfer tax.
Benchmark-level coefficients reveal positive and negative associations with multimodal performance. Unweighted text leaderboard averages can reward dimensions that transfer poorly.
These counts summarize sign stability across six robustness refits, with at least five consistent signs required for a stable classification (Appendix I). They are distinct from the individual main-fit coefficients shown in the bar chart.
A negative main-fit coefficient does not imply a stable transfer tax across refits. For example, boolean_expressions_hard is negative here but classified as unstable in Appendix I. Coefficients describe fitted associations, rather than causal effects.
Instruction tuning starts ahead.
Base models scale further.
Instruct backbones have a higher fitted transfer coefficient, but their absorption-decay coefficient Bₘ is approximately 1.33× that of Base backbones (0.0104 vs. 0.0078). In B̂ = B₀ − BₘS, this means absorption decreases more steeply as text capability S increases. Base models have a larger average absorption rate and can overtake at larger data budgets.
This is an aggregate tendency, with family-specific exceptions. Among pairs with positive slope differences, the median estimated crossover is ≈141B multimodal tokens (Appendix J).
| Fitted parameter | Base | Instruct |
|---|---|---|
| Transfer · Â | 0.212 | 0.254 |
| Absorption decay · Bₘ | 0.0078 | 0.0104 |
| Mean absorption · B̄ | 0.0161 | 0.0147 |
Families occupy different transfer–absorption regimes.
| Model family | Transfer · Â | Absorption decay · Bₘ | Mean absorption · B̄ |
|---|---|---|---|
| Llama | 0.361 | 0.0455 | 0.0122 |
| Gemma-2 | 0.324 | 0.0451 | 0.0213 |
| Falcon3 | 0.288 | 0.0133 | 0.0117 |
| Qwen3 | 0.246 | 0.0219 | 0.0126 |
Llama has the highest transfer slope and fastest absorption decay; Gemma-2 has the highest mean absorption. Backbone choice depends on the intended training regime.
From predicting performance to planning training.
The same metric guides
batch-size extrapolation.
A controlled sweep of five backbones and five batch sizes (128–2,048) finds that the optimal batch size increases with the composite capability–data metric.
Qwen3-1.7B prefers a larger batch than Falcon3-3B, reversing their parameter-count ordering.
Predicted optimal batch sizes for Qwen2.5-72B at 50B tokens (§4.4). These are extrapolations, not measured sweep optima.
Scope and limitations
The law is validated under a unified LLaVA-OneVision late-fusion recipe with a fixed SigLIP vision encoder. Generalization to early-fusion or native multimodal architectures, other training recipes, and jointly scaled vision encoders remains future work.
Choose by capability.
Plan for the data.
A quantitative bridge from language backbones
to multimodal learning.
Citation
If you find this work useful, please cite our paper.
@misc{li2026transferstextvision,
title={What Transfers from Text to Vision? Capability Scaling Laws and Transfer Dynamics for VLMs},
author={Ziran Li and Qiang Wang and Zhengyu Chen and Shanglin Lei and Borun Chen and Jingang Wang and Xunliang Cai},
year={2026},
eprint={2608.00013},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2608.00013}
}