• A Royal Holloway Dean worries that AI will stop people from asking how they know something is true.
• Lucy Gill-Simmen says AI can deliver answers, shortcutting the struggle that makes learning happen.
• "It risks becoming a substitute for thinking," she told Business Insider.

AI can produce convincing answers, but a professor says the greater risk is when people stop questioning them.

Lucy Gill-Simmen, associate dean for Education and Student Experience at Royal Holloway, University of London, said the technology could make people less willing to question, verify, and investigate information on their own.

"Hallucinations are a visible problem because we notice them when an answer is wrong," she told Business Insider of AI's well-documented tendency to invent false information. "The deeper risk is that people stop asking how they know whether an answer is right."

Gill-Simmen said knowledge has traditionally required effort — comparing sources, testing assumptions, and working through uncertainty. AI, by contrast, can produce a convincing answer without requiring users to go through that process.

"My concern is not simply misinformation," she said. "It is that people become less inclined to verify, question, and investigate for themselves. In education, the struggle is often where learning happens. AI removes much of that struggle."

AI as a substitute for thinking

Gill-Simmen's concern echoes a growing body of research into what Wharton researchers have called "cognitive surrender" — a tendency to accept AI-generated answers rather than actively evaluate them.

Steven Shaw, a postdoctoral researcher in marketing at Wharton, warned about it in a report earlier this year, telling Business Insider that people risk becoming "passive followers of unthought thoughts" by adopting AI-generated ideas without fully processing them.

In three experiments involving 1,372 participants completing 9,593 reasoning tasks, he found that participants chose to consult an AI assistant on more than half of the tasks and, once they did, accepted its recommendations about 92.7% of the time when it was correct and 79.8% of the time even when it was intentionally wrong.

Similar concerns have emerged in education. Kimberley Hardcastle, a business and marketing professor at the UK's Northumbria University, said AI could erode people's ability to independently verify, challenge, and construct knowledge without algorithms.

Gill-Simmen described the phenomenon as "epistemic atrophy," or "the gradual weakening of the habits through which knowledge is acquired."

"When people work through a problem themselves, they develop mental models, strengthen reasoning skills, and build confidence in their own thinking," she said.

"The issue isn't that AI becomes a substitute for memory. It's that it risks becoming a substitute for thinking."

'Illusion of understanding'

Gill-Simmen sees this firsthand among her students, who she said sometimes experience an "illusion of understanding."

An AI-generated explanation can sound convincing, but when students are asked to explain the reasoning themselves or apply an idea in a new context, the depth of understanding is often much weaker, she said.

Wharton professor Ethan Mollick said that AI works best as a form of "co-intelligence" that helps people explore ideas and challenge assumptions, provided humans remain responsible for evaluating its output.

In a paper he coauthored with learning scientist and wife Lilach Mollick, he wrote that students should "critically assess and interrogate AI outputs, rather than passively accept them," allowing AI to "serve as a supportive tool for their work, not a replacement."

"Ironically, in a way, the more capable AI becomes, the more important these human capacities become," Gill-Simmen said.

"In a world filled with fluent AI-generated outputs, success will depend less on producing information and more on evaluating it wisely," she added.

• In October, Sam Altman said "erotica for adults" was coming to ChatGPT.
• Now those plans are reportedly being mothballed.
• It's fine for young startups and even mature companies to try out new ideas. But OpenAI and Altman are trying out a lot.

Last fall, Sam Altman told us he was about to bring spicy chat — "erotica for adults," in his words — to ChatGPT.

That never happened, and now it looks like it never will: Altman's OpenAI has put those plans on hold "indefinitely," per the Financial Times.

This is Altman's second big walkback in the last few days. Earlier this week, the company canned Sora, the briefly popular video app it rolled out last fall. I've asked the company for comment.

Both retreats are supposed to be part of a new push at OpenAI to focus the company's efforts on things that could make money today, as it preps for an IPO at the same time it faces real competition from the likes of Google and Anthropic.

So all this starting and stopping could be viewed as necessary growing pains at a fast-growing tech company — ones that won't mean anything in the long run, if it delivers on its world-changing ambitions.

Not only that, but Altman told us we should expect this sort of stuff. "Please expect a very high rate of change from us," he wrote last fall, after hearing from content owners who were outraged to find their stuff on Sora without their permission. "We will make some good decisions and some missteps, but we will take feedback and try to fix the missteps very quickly."

It's not that companies aren't allowed to make wrong turns and head up dead ends as they grow up, and even once they're fully mature. That kind of pivoting is celebrated in tech (and is why very few people are mad that Mark Zuckerberg has stopped telling us the metaverse is the future, or that Google once bought Motorola and decided that was a bad idea a couple years later.)

But "move fast and break things" lands differently when the company doing the moving and breaking isn't running a photo app or playing around with crypto.

Instead, OpenAI and its competitors say they're leading us into a world where everything — the way we live and work (or don't work) and fight wars and everything else — will change in fundamental ways.

And investors have bought this pitch, which means our economy now seems yoked to all this — which means all of us are yoked to it, even if we never touch a chatbot.

Which makes me slightly queasy to see Sam Altman promise dirty chats in October, and then walk away from the plan less than six months later.

Not because dirty chat is obviously absurd. Lots of people in AI think romantic or sexual chatbot conversations are a real use case and could be a real business.

But the reasons it might be a bad idea for OpenAI were pretty obvious from the start. It's a giant, heavily scrutinized company that wants to be treated as central and indispensable, and it's only going to get more scrutiny.

If those objections only became real after Altman floated the idea in public, that's not charming startup experimentation. It's a sign that OpenAI is still making itself up as it goes. And that would be easier to shrug off if the rest of us weren't already being told to build our lives, jobs, and businesses around what OpenAI says comes next.