Connor Boyle

MIT Researchers Named Their Project After a Paper They Don't Seem to Have Read

by Connor Boyle

tags: science

A few days ago, I came across the paper “Demoing Stochastic Parrot: A Candid AI Cohabitant” by Chang et al. This paper—published by MIT researchers at the ACM Conference on Human Factors in Computing Systems in April of this year—presents its authors’ work on a physically embodied “AI cohabitant” that they call a “stochastic parrot”. It begins with the following passage:

AI assistants today are often built to be overly agreeable and deferential to users. This sycophantic behavior leads to AI models that agree with factually incorrect statements, reinforce users’ existing beliefs, or suppress their own reasoning to please users and avoid conflict [2].

…

[2]: Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (Virtual Event, Canada) (FAccT ’21). Association for Computing Machinery, New York, NY, USA, 610–623. doi:10.1145/3442188.3445922

I was immediately perplexed by this passage, because the cited paper by Bender and Gebru et al.—the paper that coined the term “stochastic parrot”—does not contain even one of the cited claims. Indeed, none of the words “agreeable”, “deferential”, nor “sycophantic” (nor any close synonym that I could think of) appears even once in the original “Stochastic Parrots” paper. I’m also not sure what Chang et al. mean by the phrase “AI assistants”; the original “Stochastic Parrots” paper discusses language models, not voice assistants such as Siri, Google Assistant, or Samsung Bixby. It certainly isn’t about LM-based chatbots such as ChatGPT, Claude, or Gemini; ChatGPT was not even released until November of 2022, more than a full year after the publication of “On the Dangers of Stochastic Parrots”!

The most egregious discrepancy (between Chang et al.’s statements and the source cited for them) is the claim that AI assistants “suppress their own reasoning”. Even a cursory read of “Stochastic Parrots” makes it quite clear that its authors don’t believe language models have any faculty for reasoning at all:

LMs are not performing natural language understanding (NLU), and only have success in tasks that can be approached by manipulating linguistic form.

(Bender and Gebru et al., p. 611)

Summary

Fundamentally, Bender and Gebru et al. do not argue that language models “parrot” their inference-time users (as Chang et al. seem to believe), but rather their training data. In the authors’ own words (emphasis mine):

…an LM is a system for haphazardly stitching together sequences of linguistic forms it has observed in its vast training data, according to probabilistic information about how they combine, but without any reference to meaning: a stochastic parrot.

(Bender and Gebru et al., p. 617)

“On the Dangers of Stochastic Parrots” was an immensely influential (as well as controversial) paper; as I write this, it has been cited 14,607 times according to Google Scholar, a rate of over 7.5 citations per day since it was first published. Frankly, it’s pretty bizarre to me that a group of researchers at an institution such as MIT publishing at peer-reviewed conferences such as NeurIPS would misunderstand the basic purport of a key citation like this; especially in a project named after the iconic term introduced in that paper. Yet, it appears to be an undeniable fact that they have!