Torras.ai

Meta-language refers to a language or set of terms used to describe, analyze, or discuss language itself. It involves using language to talk about language, including its structure, functions, and usage. For example, when we discuss the grammar rules of English, such as the definitions of nouns, verbs, and adjectives, we are using meta-language. A specific example of meta-language describing spelling would be explaining that the word "receive" follows the "i before e except after c" rule, as do words like "ceiling" and "deceive."

One area where AI shines is in its ability to process and generate human-like text, making it an invaluable tool for communication and information retrieval. However, large language models like ChatGPT, often struggle with meta-language tasks, such as reasoning about spelling. This limitation became evident during this recent conversation with GPT-4o.

The AI quickly provided an accurate list of the prime ministers of Spain since the restoration of democracy, which began with the transition after the death of Francisco Franco in 1975. The initial response was impressive and detailed.

However, when I asked the AI to filter the list to include only those prime ministers with a “Z” in their last name, things started to go awry. The initial response omitted some obvious candidates.

The AI provided a list but missed a few names. It’s a small error, but it highlighted how AI can sometimes miss details that a human would catch easily.

When I pointed out that Adolfo Suárez also has a “Z” in his last name, the AI quickly corrected itself. This interaction demonstrated that while AI can provide accurate information, it may need human oversight to ensure completeness.

The AI acknowledged the oversight and updated the list accordingly.

Finally, when I mentioned Pedro Sánchez, the AI once again revised its list. This iterative process shows both the strength and the current limitations of AI in handling complex queries involving meta-language.

The AI added Pedro Sánchez to the list, completing the set of names with a “Z” in their last name.

Now that GPT-4o is humbly accepting errors, let's gaslight it a little bit. "Rajoy" obviously doesn't contain the character "Z", but still, let's have some fun! 😄

The answer is surprising, to say the least. 🤔

Now we have GPT-4o completely confused and absolutely unable to solve a problem that any 5-year-old could solve.

Of course, we know why this is happening. ChatGPT uses the Transformer architecture, which is ultimately based on Tokenization, Word Embeddings and Positional Encoding. Processing back and forth from tokens to and from actual words is handled by pre and post-processing steps, so the "core" of ChatGPT doesn't really know the spelling of words.

While AI has come a long way and offers significant benefits, it’s essential to remain aware of its limitations. As demonstrated, AI can sometimes miss simple details, especially when it comes to tasks involving meta-language and reasoning about spelling.

Despite these challenges, I remain optimistic about the future of AI and its potential to improve and evolve!