This essay covers the problems with using words we use to describe human cognition to talk about LLMs, Alan Turing’s influential solution to the problem that consciousness poses for artificial intelligence, how science fiction explains why that solution no longer works, and the advantages of thinking about LLMs as a cultural technology. Of course, William James helps me pull all this together.
Many thanks for the much needed critical analysis of the language and conceptual frameworks that are being bandied about to describe LLMs. There really should be a clearer distinction between human cognition and the “computational intelligence” of LLMs.
As you point out, the terminological appropriation of human cognitive terms like “attention” and “intelligence” obscure things – especially when not even the engineers really know what is going on with these LLM computations. Hopefully, more work like Anthropic’s will shed more light on how word vectors affect LLM output. We need a more transparent understanding of the workings of these black boxes.
But on at least the superficial level of language production, there are still conceptual similarities between how humans generate much of Realtime speech and language and how LLMs generate text. There are clear parallels: both are basically associative/sequential probabilistic processes based on weighted frequencies. But there are crucial differences. I am working on a paper exploring how this doesn’t capture how humans write and think more deeply and “creatively” about things, which depends on more on hierarchical and iterative processing. I am with Yan LeCun on these limitations of LLMs and how far away they are from human cognition.
Unfortunately, I think we are a long way from even understanding human cognition and consciousness. Yet I am optimistic that generative AI advances and more transparent/explainable AI will help refine our understanding of what human consciousness and cognition are and what they are not.
Hey Nigel, Thanks for the thoughtful comment and the heads up about your own work. I just subscribed to your Substack.
I completely agree that the conceptual similarities are there and are interesting, but as you say, the "crucial differences" matter. One place to start thinking about the limits of computational explanations is the functional explanations (meaning Darwinian) that James, Mary Whiton Calkins, John Dewey, G. H. Mead, and Jessie Taft used to understand consciousness.
Many thanks for subscribing to my Substack. I just started playing this platform game a couple of weeks ago, so I appreciate your support. There are very interesting and thoughtful writers here ... especially in the gen AI niche. There seems to be a real positive community vibe. :)
I am going mull a bit more on your "functional explanations" angle, but after reading your 2-part essay, I am ready to make some revisions to my paper (which I gave an earlier version to Nick Potkalitsky who also gave me some ideas for improvement).
Nick is a treasure. Incredibly supportive of new writers in this space. Substack is an interesting set of contradictions. I am hopeful its aspirations to be a economically viable space for sharing writing are realized, sort of a wikipedia except for writing instead of basic knowledge. In addition to finding new and interesting voices, I love the distraction-free reading experience. I try not to get carried away though, recognizing it could all come crashing down when the investors start demanding a return.
Many thanks for the much needed critical analysis of the language and conceptual frameworks that are being bandied about to describe LLMs. There really should be a clearer distinction between human cognition and the “computational intelligence” of LLMs.
As you point out, the terminological appropriation of human cognitive terms like “attention” and “intelligence” obscure things – especially when not even the engineers really know what is going on with these LLM computations. Hopefully, more work like Anthropic’s will shed more light on how word vectors affect LLM output. We need a more transparent understanding of the workings of these black boxes.
But on at least the superficial level of language production, there are still conceptual similarities between how humans generate much of Realtime speech and language and how LLMs generate text. There are clear parallels: both are basically associative/sequential probabilistic processes based on weighted frequencies. But there are crucial differences. I am working on a paper exploring how this doesn’t capture how humans write and think more deeply and “creatively” about things, which depends on more on hierarchical and iterative processing. I am with Yan LeCun on these limitations of LLMs and how far away they are from human cognition.
Unfortunately, I think we are a long way from even understanding human cognition and consciousness. Yet I am optimistic that generative AI advances and more transparent/explainable AI will help refine our understanding of what human consciousness and cognition are and what they are not.
Hey Nigel, Thanks for the thoughtful comment and the heads up about your own work. I just subscribed to your Substack.
I completely agree that the conceptual similarities are there and are interesting, but as you say, the "crucial differences" matter. One place to start thinking about the limits of computational explanations is the functional explanations (meaning Darwinian) that James, Mary Whiton Calkins, John Dewey, G. H. Mead, and Jessie Taft used to understand consciousness.
Many thanks for subscribing to my Substack. I just started playing this platform game a couple of weeks ago, so I appreciate your support. There are very interesting and thoughtful writers here ... especially in the gen AI niche. There seems to be a real positive community vibe. :)
I am going mull a bit more on your "functional explanations" angle, but after reading your 2-part essay, I am ready to make some revisions to my paper (which I gave an earlier version to Nick Potkalitsky who also gave me some ideas for improvement).
Thanks again.
Nick is a treasure. Incredibly supportive of new writers in this space. Substack is an interesting set of contradictions. I am hopeful its aspirations to be a economically viable space for sharing writing are realized, sort of a wikipedia except for writing instead of basic knowledge. In addition to finding new and interesting voices, I love the distraction-free reading experience. I try not to get carried away though, recognizing it could all come crashing down when the investors start demanding a return.