Can One LLM Win at Everything? Finding the Pareto Frontier

01 / MOTIVATION & TASK

Chatbot Arena turns pairwise preferences into model rankings. But a ranking alone does not settle which model to use when we care about several capabilities. A model that leads in math may fall behind in coding, and a single overall score depends on how we weight those capabilities.

Before choosing those weights, we can ask a more basic question: which models can we rule out because another model is better on every objective? The remaining models form the strict Pareto frontier.

Pareto identification from pairwise feedback

We use strict dominance: model $$j$$ dominates model $$i$$ only if its latent score is larger in every objective. The strict Pareto frontier consists of the models with no such dominator. This differs from the common convention of “at least as good everywhere and strictly better somewhere.” Since a user may care only about an objective on which two models perform equally well, neither should be discarded on performance grounds: token cost, availability, or other application-specific considerations may instead determine the final choice.

Human-preference judgments are costly to collect, and evaluating several capabilities requires enough evidence for each objective. We seek to identify this frontier without first estimating all latent scores or pairwise win probabilities. Our approach uses adaptive sampling to progressively classify models whose Pareto status can be certified and concentrate comparisons on those that remain difficult to classify. Therefore, we can reduce the sampling cost compared with uniform sampling.

Our objective is fixed-confidence identification: recover the exact strict frontier with probability at least $1-\delta$ , while minimizing the number of objective-specific comparisons.

02 / Try the frontier

Try the frontier in 2D or 3D

Select a dataset and two or three objectives to inspect the empirical frontier. A coral ring marks frontier membership in the selected projection; the full-dimensional target uses all objectives. Coordinates are empirical Vector-Borda scores, not fitted latent scores.

Projection frontier—

Full-dimensional frontier—

Roster / objectives—

—

Left-drag to move the view · mouse wheel to zoom

Preparing the interactive frontier…

03 / Core idea

Vector-Borda reduction and adaptive elimination

Consider $$K$$ models and $$d$$ objectives. Each query selects a model pair and one objective, and returns a binary preference. Under the Bradley–Terry (BT) model, the win probability is $\sigma(\theta_{i,r}-\theta_{j,r})$ , where $\theta_{i,r}$ is model $$i$$ ’s latent score and $\sigma(x)=(1+e^{-x})^{-1}$ .

We do not need to estimate every pairwise win probability to identify the strict Pareto frontier. Instead, the Vector-Borda reduction lets us estimate each model’s win probability against a uniformly random opponent on each objective— $$Kd$$ scores in total. Under BT, these scores preserve the latent ordering on every objective.

Estimate every pair$\Theta(K^2d)$

Pair–objective quantities to reconstruct.

Estimate one score stream$\Theta(Kd)$

Random-opponent means to learn.

For model $$i$$ and objective $$r$$ , define its coordinate-wise Borda score as the probability that it beats a uniformly random opponent:

b_{i,r}=\frac{1}{K-1}\sum_{j\ne i}\sigma\!\left(\theta_{i,r}-\theta_{j,r}\right)

Choose a focal model and an objective, draw an opponent uniformly from the other models in the original roster, and record the outcome. This produces a Bernoulli observation with mean $b_{i,r}$ . Building on random-opponent Borda sampling in dueling bandits, we apply the reduction coordinate-wise to preserve a multi-objective target.

Order preservation

\operatorname{sign}\!\left(b_{i,r}-b_{j,r}\right)=\operatorname{sign}\!\left(\theta_{i,r}-\theta_{j,r}\right)

The intuition is simple: under the BT model, a model with stronger math ability has a higher chance of beating any given opponent at math. Its average win rate is therefore higher too. This holds for every objective, so using average win rates instead of latent abilities leaves the strict Pareto frontier unchanged.

How VB-EGE allocates comparisons

We therefore use this reduction to build Vector-Borda Empirical-Gap Elimination (VB-EGE), adapting the empirical-gap approach of Kone et al. (2024). In each phase, we collect more comparisons for the models still under consideration. As the estimates become more precise, we decide which models belong to the frontier and which can be ruled out.

Sampling phase / collect evidence

Collect more comparisons.

Bring every active model–objective pair to the same sample count for this phase. Opponents are drawn from the full original roster, including models already classified.

For a dominated model, the classification gap measures the strongest evidence that another model exceeds it in every objective. Confirming that a model is on the frontier is not always enough to stop sampling it. We may still need a more accurate estimate of its scores to rule out another model. The classification gap accounts for both tasks.

Within a phase, VB-EGE selects the largest empirical gap exceeding $$4r_m$$ , where $$r_m$$ is the current confidence radius. It accepts an empirically non-dominated model or rejects an empirically dominated one, then recomputes the remaining gaps using the same observations. New samples are collected only when no further removal is certified.

Two scatter plots showing that models with smaller Borda classification gaps receive more VB-EGE samples. — **Allocation by classification gap.** On representative Arena-4 medium and Two-group-10 instances, smaller-gap models receive more comparisons. The vertical axes are logarithmic (paper, Figure 6).

Fixed-confidence and minimax guarantees

Under the BT model and positive classification gaps, VB-EGE returns the exact strict frontier with probability at least $1-\delta$ . Its high-probability comparison bound scales with $d\sum_i(\Delta_i^B)^{-2}$ , up to logarithmic factors, where $\Delta_i^B$ is model $$i$$ ’s Borda classification gap. Small-gap models therefore determine the cost.

For a fixed bounded latent-score range and minimum latent classification gap $\Delta$ , the worst-case stopping-time quantile scales as $\widetilde{\Theta}\!\left(dK\Delta^{-2}\log(1/\delta)\right)$ as $\delta\to0$ . The upper and lower bounds match up to logarithmic factors, establishing minimax optimality in the high-probability stopping-time sense.

04 / Evidence

What the experiments show

We test whether the algorithms recover the exact strict frontier, their mean stopping time, and how this cost scales. Synthetic BT instances let us vary the number of models, objectives, and classification gaps across different settings. We also replay three Arena datasets to evaluate comparison efficiency on logged human preferences.

We compare against three uniform-sampling baselines: Focal Borda without elimination, BT-MLE using fitted latent scores, and a Borda plug-in method using estimated pairwise win rates. Detailed experimental settings are provided in the paper.

500trials per main synthetic setting

1.09descriptive log–log slope in both

\(K\)

and

\(d\)

100/100exact recoveries per method and real dataset

No set errors were observed in these experiments. Finite-run recovery rates do not establish zero error probability, and not every baseline has a fixed-confidence guarantee.

Lower is better / logarithmic axis

Real-data replay cost

Grouped bar chart comparing mean coordinate-specific comparisons on three fixed Arena datasets; VB-EGE is lowest in each group. — Mean stopping time over 100 independent replays of each dataset. VB-EGE has the lowest mean stopping time on all three datasets. The vertical axis is logarithmic. Error bars show one standard error.

Scaling and heterogeneous benchmarks

Mean stopping time scales roughly linearly with the number of models and objectives, decreases with larger gaps, and grows with $\log(1/\delta)$ , consistent with theory. Across eight heterogeneous benchmarks, VB-EGE has the lowest mean stopping time. Focal Borda requires 1.25–111× as many comparisons, measured by the ratio of mean stopping times.

Replaying fixed Arena logs

All methods recover the empirical frontier in all 100 replays per dataset. VB-EGE has the lowest mean stopping time: Focal Borda’s mean cost is 16.54×, 3.96×, and 4.59× higher on Arena-35, SearchArena-8, and VisionArena-12, respectively.

05 / Future work

Future directions

Several directions remain open, both in modeling Arena feedback and in designing sampling strategies.

Beyond Bradley–Terry comparisons

The Borda reduction preserves order whenever win probability is strictly increasing in the latent-score difference. This suggests extending our guarantees beyond BT to other monotone comparison models.

Ties and “both bad” feedback

Our appendix studies half-credit treatments of ties and “both bad” votes and shows that the Borda-level recovery guarantee extends to direct half-credit observations under positive classification gaps. A fuller model of these distinct feedback types remains future work.

Objective labeling

Our datasets group prompts in different ways. How sensitive is the frontier to these labels? Soft or learned assignments could let a comparison contribute to several capabilities rather than a single category.

Adaptive likelihood-based methods

Another approach is to estimate latent scores with MLE and use those estimates to guide sampling. Our appendix explores a Pareto-specific heuristic inspired by the scalar top-k Track-and-Stop method of Goldberger and Rudi (2026). Developing a method with rigorous guarantees remains future work.

Sparse and evolving comparison graphs

Our algorithm assumes a complete comparison graph: every model pair can be compared on every objective. Real Arena logs often lack some of these comparisons. Designing algorithms that identify the frontier reliably with missing comparison data remains an open problem, especially as new models enter the arena.

06 / References

References and related work

01
Chiang et al. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. Pairwise human-preference data and Arena evaluation.
02
Liang et al. (2023). Holistic Evaluation of Language Models. Multi-metric, multi-scenario evaluation.
03
Jamieson et al. (2015). Sparse Dueling Bandits. Random-opponent Borda sampling for pairwise comparisons.
04
Auer et al. (2016). Pareto Front Identification from Stochastic Bandit Feedback. Early confidence-based Pareto identification.
05
Kone, Kaufmann, and Richert (2024). Bandit Pareto Set Identification: the Fixed Budget Setting. Empirical-Gap Elimination for vector observations.
06
Crépon, Garivier, and Koolen (2024). Sequential Learning of the Pareto Front for Multi-objective Bandits. Exact fixed-confidence Pareto learning with asymptotically optimal allocation.
07
Goldberger and Rudi (2026). Optimal Top-k Identification from Pairwise Comparisons. Adaptive MLE and Track-and-Stop for the scalar top-k target.
08
Miroyan et al. (2026). Search Arena: Analyzing Search-Augmented LLMs. Source and documentation for Search Arena 24K.
09
Chou et al. (2025). VisionArena: 230K Real World User-VLM Conversations with Preference Labels. Source and documentation for VisionArena-Battle.

How to cite

The paper can be cited as follows.

Bibliography

Liang Twist Shan, Jiahao Ai, and Zhimei Ren. “Beyond Scalar Leaderboards: Adaptive Sampling for Pareto Frontier Identification in Multi-Objective LLM Arenas.” arXiv preprint arXiv:2610.27447, 2026.

BibTeX

@misc{shan2026beyondscalarleaderboards,
  title = {Beyond Scalar Leaderboards: Adaptive Sampling for
           Pareto Frontier Identification in Multi-Objective {LLM} Arenas},
  author = {Shan, Liang Twist and Ai, Jiahao and Ren, Zhimei},
  year = {2026},
  eprint = {2610.27447},
  archivePrefix = {arXiv},
  primaryClass = {stat.ML},
  url = {https://arxiv.org/abs/2610.27447}
}