Eliza in the Boardroom: Why a Successful AI Demo Doesn't Mean You're Ready for AI Transformation

What a system says doesn't tell you how it works. Your organization makes the exact same mistake when it confuses an impressive demo with readiness for AI transformation.

Key takeaways

• Output is not the mechanism. The fact that a result looks human doesn't mean it was produced the way a human would produce it — the core argument from Gary Marcus and co-authors in Nature (February 2026).
• Boards make the same mistake as AI enthusiasts, just in reverse: they look at an impressive pilot demo and infer readiness for the entire organization.
• The numbers are brutal: roughly 95% of generative-AI pilots produce no measurable return in P&L (MIT NANDA, 2025). Only 5–6% of companies actually capture value (BCG, McKinsey, 2025).
• AI transformation doesn't break on technology. It breaks on the layer a demo never shows: trust between people, the quality of information flow, and the alignment of goals across the organization.
• AI readiness can't be declared — it has to be measured. That's exactly what Collaboration.tech does.

A debate that looks academic — and costs millions

In late May 2026, two voices collided around a single question: do large language models actually understand?

On one side, Geoffrey Hinton — Nobel laureate in physics (2024) and one of the fathers of deep learning — has for some time suggested that today's AI models possess something like understanding, even inner states resembling those of humans.

On the other side, Pope Leo XIV took a very different position in his first encyclical, Magnifica Humanitas (released 25 May 2026). AI systems, the Pope writes, "may imitate… or even simulate… but they do not understand what they produce," because they lack the affective, relational, and spiritual perspective through which a human being grows in wisdom. Real understanding comes from experience, not from statistical adaptation to data.

AI critic Gary Marcus summed it up acidly but accurately in an essay whose very title reads: the Pope appears to understand AI better than Geoffrey Hinton does. The subtitle captures the whole idea: what a thing says doesn't tell you how it came to say it.

The dispute looks philosophical. It isn't. Beneath it sits a cognitive error that costs organizations real money every month — committed not against the model, but against themselves.

Output is not the mechanism — the heart of the argument

A language model produces fluent, human-sounding text because it was trained to predict the next word from vast datasets — not because it lived through something and understood it. Two systems with very similar outputs can arrive at them in radically different ways.

Marcus, together with Walter Quattrociocchi and Valerio Capraro, described this in Nature (the comment "Statistical approximation is not general intelligence," 17 February 2026) as a fundamental fault line in how we evaluate AI. The dominant assumption runs: "if a system statistically approximates human behavior, then it is close to a human." That assumption is false. Success in behavioral tests — including variants of the Turing test — is not evidence of general intelligence, nor of understanding.

This is a modern version of the Eliza illusion: back in 1966, the simple ELIZA program, which merely rephrased the user's sentences as questions, made people feel they were talking to someone who understood them. We see fluent behavior and project a depth that isn't there.

Your board makes the same mistake

Now let's move this from the model to the organization.

A company launches a pilot. An AI assistant writes great emails, summarizes reports, generates graphics, suggests the next step. The demo impresses the board. The conclusion lands: "It works. We're ready for AI transformation."

That is exactly the same mistake as Hinton's — just mirrored. Hinton looks at the model's impressive output and infers something about its interior. The board looks at the pilot's impressive output and infers something about the readiness of its own organization. In both cases they confuse the effect with the mechanism that produces it.

A successful demo says nothing about how it came to be:

• whether it worked thanks to organizational maturity, or in spite of its absence — because one enthusiast was patching everything by hand;
• whether it can be repeated at scale, or whether it will fall apart with the third team;
• whether the people who are meant to use it day to day trust one another enough to share data and work.

To borrow a manufacturing metaphor: most companies that "do something with AI" have swapped out the engine but never rebuilt the production line. A generic assistant handed to the whole company can deliver 80% adoption and 0% impact on transformation. People start writing faster emails — and that's it. Organizational productivity doesn't move.

What you really don't see in a demo

The data is clear. According to the MIT NANDA report from 2025 (a sample of about 300 deployments), roughly 95% of corporate generative-AI pilots produce no measurable return in the income statement (P&L). This is not an isolated reading — three independent 2025 studies paint the same picture: MIT NANDA 2025 (~300 deployments) found 95% of GenAI pilots with no measurable P&L return; BCG's Build for the Future 2025 (2,000 firms) identified just 5% as "future-built," deeply integrated with AI; McKinsey's State of AI (November 2025) found only 6% qualify as "high performers" generating more than 5% EBIT from AI.

Three samples, one result: only five to six percent of companies capture real value from AI.

The reason is mundane and uncomfortable. AI transformation doesn't break on technology. When AI leaders honestly assess the maturity of their own organizations, the weakest dimension turns out to be change management, with strategy and governance close behind. Technology, data, and talent are "okay-ish." The problem is that nobody designs the transformation — and so is everything happening underneath: trust between teams, the quality of information flow, willingness to share data, and the actual — not the declared — patterns of working together.

You don't see this in a demo. You don't see it with the naked eye at all.

And here we return to Pope Leo XIV's encyclical. His sentence — real understanding comes from experience, not from statistical approximation — describes organizations too. A company that buys AI tools without the collaboration infrastructure underneath isn't undergoing a transformation. It is approximating one. Simulating readiness. Imitating change. Just as the model imitates understanding.

First, a distinction: AI is a tool; collaboration is a process between people

In this whole discussion it's easy to fall for a mental shortcut that is itself a trap: "human–AI collaboration." It's a convenient metaphor, but a misleading one. Collaboration is a social process — it occurs only between people (teams, departments, organizations) and consists in generating alignment between goals across different levels.

AI does not collaborate. AI is a tool that people use. This distinction isn't pedantic — it has practical consequences. If the layer of collaboration between people is weak — team goals are misaligned, trust is low, information doesn't circulate — then no AI tool will fix it. Worse, the tool will only expose and accelerate the existing chaos.

That's why the real foundation of AI transformation is not the model itself. It's the alignment of goals and the quality of collaboration among the people who use that model. This is the layer that must be diagnosed before you start scaling the technology.

Postulating versus measuring

Most of the advisory market stops at the postulate: "you need to build an AI-ready culture," "you need trust and collaboration." True — and completely useless until it answers the question: ready to what degree, exactly where, and what do we do about it on Monday morning.

You can't manage what you haven't measured. And organizational readiness for AI can't be read off the enthusiasm in a workshop or a single successful pilot — just as a model's understanding can't be read off one polished answer. You have to look underneath, at the mechanism.

That is exactly what Collaboration.tech does. Instead of asking whether AI "works" in a demo, we measure what truly decides the success of a transformation: the real state of collaboration, trust, information flow, and goal alignment in the organization — the layer no demo will ever reveal. Diagnosis instead of declaration. Mechanism instead of effect.

The bottom line: ask the question the Pope would ask, not Hinton

Hinton looked at what the model says and filled in an interior. Let that be a warning, not a template.

Before your organization decides it's ready for AI because a demo impressed everyone, ask the question the Pope would ask, not Hinton:

Does what I'm seeing actually tell me how it works underneath?

If you're not sure — measure it. Because focus always beats dispersion, and diagnosis always beats declaration.

FAQ — common questions about AI transformation readiness

Why do most AI pilots fail to deliver value?

Because a successful demo says nothing about how it was produced or whether it can be repeated at scale. According to MIT NANDA (2025), about 95% of generative-AI pilots produce no measurable P&L return. The main cause isn't technology — it's weak change management and the absence of a collaboration infrastructure between people.

What is organizational AI readiness?

It's the degree to which an organization is actually able to turn AI tools into value. It comprises, among other things, alignment of goals between teams, trust, the quality of information flow, willingness to share data, and a designated transformation owner with a mandate from the board. Readiness can't be declared — it has to be measured.

Does a human "collaborate" with AI?

Not in the strict sense. Collaboration is a social process that takes place between people and rests on generating alignment of goals. AI is a tool that people use. If collaboration between people is weak, an AI tool won't replace it — at best it will expose and accelerate existing problems.

What did Pope Leo XIV say about artificial intelligence?

In the encyclical Magnifica Humanitas (25 May 2026), he wrote that AI systems "may imitate… or even simulate… but they do not understand what they produce," because they lack the affective, relational, and spiritual perspective. Real understanding comes from experience, not from statistical adaptation to data.

Where should a company start with AI transformation?

With a diagnosis of the layer a demo doesn't show: the alignment of goals, trust, and information flow between people. Only on that basis is it worth selecting and scaling specific AI use cases — focusing on the few most important ones rather than scattering effort across dozens of pilots.

Sources

• Gary Marcus, The Pope Appears to Understand AI Better Than Geoffrey Hinton Does — garymarcus.substack.com (May/June 2026).
• W. Quattrociocchi, V. Capraro, G. Marcus, Statistical approximation is not general intelligence, Nature 650(8102):792, 17 February 2026, DOI: 10.1038/d41586-026-00495-y.
• Pope Leo XIV, encyclical Magnifica Humanitas (released 25 May 2026) — Vatican News, TIME, NPR.
• MIT NANDA, 2025 report (sample of ≈300 deployments) — 95% of GenAI pilots with no measurable P&L return.
• BCG, Build for the Future, 2025 (2,000 firms); McKinsey, State of AI, November 2025.