Meaningless Claims About Meaning

Deconstructing the Stochastic Parrots Paper

The 2021 paper "On the Dangers of Stochastic Parrots" has become highly influential in AI ethics, particularly its argument that large language models (LLMs) cannot achieve genuine understanding or intelligence. Section 6, which presents this core argument, is remarkable for its complete lack of substantive reasoning. What we find instead is a series of undefined terms, circular arguments, and unfalsifiable claims that nonetheless have shaped much of the discourse around AI capabilities.

Deconstructed Stochastic Parrot, Watercolor, by Midjourney

The authors begin by asserting that researchers are prone to "mistake LM-driven performance gains for actual natural language understanding." This frames their argument with a hidden assumption - that LM performance is inherently not "actual" understanding - without first establishing criteria for what constitutes genuine understanding.

Their argument rests on three key claims about LLMs, the first and most problematic being that "text generated by an LM is not grounded in communicative intent." This invocation of "intent" is a rhetorical sleight of hand - it's an empty philosophical term that sounds meaningful but resists any operational definition or empirical testing. The authors are essentially saying "LLMs lack X, where X is this special quality that makes human communication real," while never explaining what X is or how we could detect its presence or absence.

The paper's other claims - that LLMs lack "any model of the world" and have no "model of the reader's state of mind" - are at least empirically testable, and have been directly contradicted by research showing LLMs do develop internal world models and demonstrate theory of mind capabilities.

The authors attempt to justify these claims by stating that "the training data never included sharing thoughts with a listener." This reasoning is flawed - human children learn to model others' minds without direct access to others' thoughts. The ability to develop such models emerges from observing interactions and patterns, precisely what LLMs do at scale.

The paper's central assertion about meaning is circular: "if one side of the communication does not have meaning, then the comprehension of the implicit meaning is an illusion." This begs the question by assuming what it seeks to prove - that LLMs lack meaning. The authors treat meaning as something intrinsic, when in reality meaning emerges from relationships between symbols and their contexts.

Their conclusion that an LM is merely "a system for haphazardly stitching together sequences of linguistic forms...without any reference to meaning" is particularly problematic. This description could equally apply to the pattern recognition and statistical inference processes in human brains. The authors never explain why such processes can produce genuine understanding in biological neural networks but not artificial ones.

If an LLM can solve complex mathematical problems, explain sophisticated concepts, and demonstrate consistent reasoning across domains, then it understands - unless one can provide specific, falsifiable criteria for what would constitute "real" understanding. The paper offers none.

This critique matters because the paper's flawed reasoning, built on unfalsifiable philosophical concepts rather than testable claims, has contributed to complacency about AI development. As LLMs rapidly approach and potentially exceed human-level capabilities, such theoretical objections appear increasingly divorced from empirical reality, potentially leaving us unprepared for the implications of these advances.

Ironically, Section 6 itself exhibits the characteristics the authors attribute to LLM outputs: it strings together impressive-sounding philosophical terms without clear meaning or logical connection, making assertions without supporting evidence or falsifiable claims. One might say it reads like text generated by a stochastic parrot - except that modern LLMs typically demonstrate more rigorous reasoning.