Estimating Consumer Preferences for LLMs: Evidence from LMArena

About this event

This paper studies product differentiation in the consumer market for large language models (LLMs) using data from randomized pairwise comparisons of LLM responses. I estimate a differentiated-product demand model in which consumer taste parameters vary with prompt embeddings. Perceived LLM quality is strongly associated with an LLM's benchmark performance, context window size, and reasoning ability. Taste heterogeneity across consumers is significant and arises primarily over the stylistic and syntactic features of responses (e.g., readability, verbosity, tone, and formatting), along which LLMs systematically differ. Counterfactual analysis shows that, between 2024 and 2025, consumer surplus per user increased by 38%-54% (equivalently, $37.4-$52.9 per user-month). For the average user, improvements in model intelligence account for 61% of this welfare gain, with an even larger share for technical tasks (71% for coding).