Gödel Escher Bach, Self-Reference, and Transformers

August, 15 2023 • 8 min read • 1300 words

Gödel's Incompleteness Theorem, Strange Loops, and Transformer Langauge Models.

Gödel Escher Bach and Strange Loops

Not too long ago, I was recommended the book “Gödel, Escher, Bach” by Douglas Hofstadter. The book, at its core, is about how intelligent systems arise out of nothing, or in the words of the author,

GEB was inspired by my long-held conviction that the “strange loop” notion holds the key to unraveling the mystery that we conscious beings call “being” or “consciousness.”

Below is one of the main subjects of the book:

The Godelian strange loop that arises in formal systems in mathematics (i.e., collections of rules for churning out an endless series of mathematical truths solely by mechanical symbol-shunting without any regard to meanings or ideas hidden in the shapes being manipulated) is a loop that allows such a system to “perceive itself”, to talk about itself, to become “self-aware”, and in a sense it would not be going too far to say that by virtue of having such a loop, a formal system acquires a self.

This Godelian strange loop he is referring to is Godel’s incompleteness theorem, which uses self-referencing meta-mathematical statements to prove that within any consistent formal system that is capable of arithmetic, there will be true statements that cannot be proved within the system. These theorems bring to light the inherent limitations of formal mathematical systems and shook the mathematical world’s understanding of truth and consistency in formal systems.

Gödel’s proof creates a mathematical statement that refers to itself, similar to the paradoxical statement “This sentence is false”, and it is this kind of self-reference that Hofstadter refers to. Hofstadter sees this self-referential quality in Gödel’s theorems as analogous to the nature of consciousness. A conscious being is able to reflect upon itself, perceive itself, and model its thoughts and thinking processes. This self-awareness can be seen as a form of a strange loop, where the mind simultaneously stands above and below itself in a hierarchy, observing and being observed, thinking and being thought about.

For Hofstadter, the strange loops that arise in Gödel’s theorems are more than mere mathematical curiosities; they serve as a metaphor or model for understanding how consciousness arises. The self-referential structures in mathematics are seen as echoing the self-referential nature of thought, where the mind can contemplate itself and create an abstract representation of its own processes.

As he said in 1979,

“It is an inherent property of intelligence that it can jump out of the task which it is performing, and survey what it has done; it is always looking for, and often finding, patterns. Now I said that an intelligence can jump out of its task, but that does not mean that it always will. However, a little prompting will often suffice.”

According to Hofstadter, self-reference and recursion can create complex, ‘intelligent’ systems. If a system continually references and interacts with itself in increasingly complex ways, it can give rise to novel patterns and behaviors. As Hofstadter puts it, “Meaningless symbols acquire meaning despite themselves”.

“Now sophisticated operating systems carry out similar traffic-handling and level-switching operations with respect to users and their programs. It is virtually certain that there are somewhat parallel things that take place in the brain: handling of many stimuli at the same time; decisions of what should have priority over what and for how long; instantaneous “interrupts” caused by emergencies or other unexpected occurrences; and so on.”

At the base level, individual simple dumb processes interacting at a large scale in the human brain have the emergent property of consciousness. We have seen emergent properties in LLMs. The added ability to self-reference and operate recursively might be a necessary property to be able to reflect upon one’s actions to act as an intelligent agent and modify one’s behavior or even code to adapt to new circumstances.

We have already seen this in the form of chain-of-thought prompting, where the output of the model is then passed in again for the model to reflect upon, and we have seen this have great results. But there seems to always be a limit of intelligence we can squeeze from these systems that lack the agency of humans.

The Limits of LLM Agents

Large language model agents emerge from complex interactions between it and vector databases that act as a memory store for previous output. This database allows the LLM to incrementally build upon prior knowledge, creating an iterative learning process that aids in generating novel and relevant responses. This, in essence, enables LLMs to “learn” and respond to new information in a contextually appropriate manner. Despite all the hype and desire for intelligent agents, these systems have seemed to always fall short. There always seemed to be some ingredient missing, some elixir of life and intelligence that we have but can’t quantify.

“No one knows where the borderline between non-intelligent behavior and intelligent behavior lies; in fact, to suggest that a sharp borderline exists is probably silly.”

Before a system can have a theory of mind, which is defined as “the ability to understand other people’s minds: mental states, knowledge, intentions, and beliefs”, one must understand their own. A system that doesn’t understand its own workings is inherently capped in the complex actions it can take. By not comprehending the intricacies of the system they operate in or that they are comprised of, these LLMs are restricted in their ability to optimize their responses or troubleshoot their processes. An LLM that doesn’t understand its own behavior can only perform tasks to the extent of its pre-defined capabilities. Humanity has created a cornucopia of complexities and truths that have been solved. From quantum mechanics and topology to architecture and art, have we not great things to show? While we might not fully understand the origin of consciousness, or be able to modify ourselves to amplify our virtues and mute our vices (yet), it is our understanding of the flaws and passion of man that helped push us forward. The Enlightenment and beyond consists of people creating new systems, either as institutions or individual frameworks, that attempt to raise our higher selves and diminish our lower instincts. It is this gentle nudge on human nature which we have seen society progress and bear fruit.

Dario Amodei, the CEO and co-founder of Anthropic, has said that a true landmark will be achieved when LLMs are smart enough to do LLM research. That would be a strange loop and require these systems to have a high understanding of their own properties and behavior.

Self-Improving Machines

“The flexibility of intelligence comes from the enormous number of different rules, and levels of rules… Strange Loops involving rules that change themselves, directly or indirectly, are at the core of intelligence”

The concept of the singularity, as originally coined by John von Neumann and later popularized by Ray Kurzweil, is intrinsically tied to the idea of self-improving intelligent machines. In “The Singularity is Near”, Kurzweil predicts that the singularity will occur around the year 2045, marking a point in time when technological growth becomes uncontrollable and irreversible, resulting in unforeseeable changes to human civilization. He argues that the rate of technological progress is exponential, and as AI begins to surpass human intelligence, it will have the ability to recursively improve its own design, leading to an intelligence explosion.

Whether we seem to hit a local maximum, or whether our current methodology of scale continues to unlock new abilities, only time can tell. As remarkable as ChatGPT and similar AI models are, they currently remain tools and are far from true intelligent systems. While the age of self-improving machines might not be immediately around the corner, the advances in transformer models have had a large impact on the societal perception of the distance we have to go. It is certain though that one day (and perhaps relatively soon) we will hit this milestone of human progress and ourselves become Prometheus or Frankenstein. May we have a better fate.