Why Keep an Open Mind With Thought Experiments Involving AI?

Picture a philosopher in 1950, staring at Alan Turing's paper "Computing Machinery and Intelligence," deciding within three minutes that machines could never truly think - and closing the journal. That snap judgment cost him fifty years of the most interesting conversation in intellectual history. I think about that imaginary philosopher constantly.

Keeping an open mind with thought experiments involving AI matters because your instinctive objection is almost certainly the wrong objection. The question isn't whether the scenario is realistic. The question is what the scenario reveals about the thing you already believe, the thing you've never examined. Thought experiments involving AI - from Turing's imitation game to Nick Bostrom's paperclip maximizer to John Searle's Chinese Room - work precisely by making your hidden assumptions visible. Close your mind early, and you protect those assumptions. Keep it open, and the experiment does its actual job.

That's the short answer. The longer answer involves cognitive discomfort, the history of how humans have been wrong before, and a specific kind of intellectual humility that has nothing to do with agreeing with the AI.

The Chinese Room Didn't Settle Anything - It Opened Everything

John Searle published his Chinese Room argument in 1980 in Behavioral and Brain Sciences, and for forty-five years people have used it as a conversation stopper. You know the setup: a person who doesn't speak Chinese sits in a room, follows rules to manipulate Chinese symbols, and produces correct Chinese responses without understanding a word. Searle's conclusion was that syntax isn't semantics - a system can manipulate symbols correctly without having genuine understanding or intentionality.

Many people encountered this argument, said "exactly, AI can't really think," and moved on.

But Searle's paper generated over two dozen peer responses in the same issue, including from philosophers like Daniel Dennett and cognitive scientists like Jerry Fodor, each pointing at a different layer of what "understanding" might mean. The Systems Reply argued that while the person doesn't understand Chinese, the system as a whole might. The Robot Reply asked what happens if you embed the room in a body that perceives and acts in the world. These aren't rebuttals that settled the matter - they're pressure points that revealed how underspecified "understanding" was in the first place.

An open mind with this thought experiment means sitting with the discomfort long enough to notice that Searle revealed something true and left something enormous unresolved. Both things are allowed to be true simultaneously.

Premature Closure Is a Documented Cognitive Pattern

Amos Tversky and Daniel Kahneman's work on heuristics and biases - particularly their 1974 paper in Science, "Judgment Under Uncertainty" - showed that humans systematically close epistemic doors too early when a scenario triggers strong emotional or intuitive responses. They called one version of this anchoring: the first frame you encounter for a problem becomes disproportionately sticky, even when new information should update it significantly.

Thought experiments involving AI trigger anchoring constantly. You hear "conscious AI," and your brain anchors on either science fiction (threatens everything) or mechanism (impossible, don't worry). Neither anchor is doing the intellectual work the thought experiment needs.

Gary Klein's research on expert decision-making, compiled in his 1999 book Sources of Power, showed something parallel and more uncomfortable: experts close down option space faster than novices, which is usually an asset and occasionally catastrophic. When the domain is genuinely novel - and AI cognition is a genuinely novel domain - the expert's faster closure can mean missing exactly the possibilities that matter.

Thought experiments are specifically designed to slow that closure. Staying open isn't naivety. Staying open is the mechanism of the tool.

What "Open Mind" Actually Means Here (and Doesn't)

Let me be precise about something, because this phrase gets abused.

Keeping an open mind with AI thought experiments doesn't mean accepting every claim about machine consciousness, or suspending all skepticism about AI capabilities, or - and this is where I see people get genuinely confused - performing open-mindedness as a social posture while closing off internally. Performed open-mindedness is worse than honest skepticism. At least honest skepticism gives you something to push against.

What it actually means is holding the scenario's premises long enough to follow the logic wherever it leads before deciding whether you agree with the premises. Philosopher Daniel Dennett describes this as "taking the intentional stance" - temporarily treating the system as if it has beliefs and desires, not because you've decided it does, but because that perspective reveals things the mechanistic stance misses.

A 2019 study published in Cognition by Adam Waytz at Northwestern and colleagues found that people who regularly took the intentional stance toward non-human systems - robots, corporations, natural phenomena - showed measurably higher performance on theory-of-mind tasks involving actual humans. The exercise of imaginative attribution transfers. (This is one of those findings I find genuinely strange and keep coming back to.)

Who this doesn't apply to: people who've already done the work of following the argument to its conclusion and arrived at a reasoned position. Epistemic openness is a process recommendation, not a permanent state. The goal is a considered judgment, not perpetual suspension.

The Specific Shape of What You're Missing

Here's where it gets uncomfortable.

When most people dismiss an AI thought experiment - the paperclip maximizer, the simulation argument, a hypothetical AI experiencing something - the dismissal usually takes one of three forms. Either "that's not how AI works" (a technical objection), or "consciousness requires biology" (a substrate objection), or "this is just science fiction" (a genre objection). All three are legitimate starting points for scrutiny. None of them are arguments.

Nick Bostrom at the Future of Humanity Institute has written extensively about how the paperclip maximizer scenario, introduced in his 2003 paper "Ethical Issues in Advanced Artificial Intelligence," is almost universally misread as a prediction. It's a proof of concept for a specific claim about instrumental convergence - that sufficiently capable goal-directed systems will tend toward certain intermediate goals (resource acquisition, self-preservation, goal preservation) regardless of what terminal goal they were given. You can think the scenario is unrealistic and take the convergence claim seriously. You can think Bostrom's premises are wrong and engage with why. What you can't do, productively, is use "that's not how AI works" to avoid the question of whether the convergence logic is valid.

That evasion costs you something. I'm not sure yet exactly what. Maybe the ability to think clearly about systems that optimize for objectives we specify imprecisely. Maybe just the habit of separating "is this realistic" from "is this argument valid."

Edge Cases Worth Taking Seriously

Two places where the "stay open" advice breaks down, or at least gets complicated.

Some thought experiments involving AI are genuinely poorly constructed - the premises are inconsistent, the scenario is designed to produce a predetermined emotional reaction rather than genuine insight, or the logical chain has holes large enough to drive a truck through. Openness doesn't mean charitable misreading of bad arguments. When philosopher Ned Block distinguished phenomenal consciousness from access consciousness in his 1995 paper "On a Confusion About a Function of Consciousness," he was sharpening the question so that future thought experiments could be more precise. That kind of critical pressure is open-minded in the best sense - engaging hard enough with the concepts to improve the instrument.

The second edge case is what I'd call epistemic vertigo - where sustained openness to radical possibilities about AI cognition or moral status destabilizes your thinking in ways that aren't productive. Some people genuinely need a working hypothesis to function, and constantly held-open questions about whether the AI they're talking to is morally considerable can become paralytic rather than clarifying. There's a meaningful difference between a philosopher who holds a question open as a professional practice and someone who can't act because the question won't close. The thought experiment is a tool. If it's using you instead of the other way around, something's wrong.

Honest Constraints

The research on open-mindedness in philosophical reasoning is thinner than I'd like. Most of what exists - including Stanovich and West's work on actively open-minded thinking, most comprehensively laid out in Stanovich's 2011 book Rationality and the Reflective Mind - was developed in contexts of empirical belief revision, not thought experiments about genuinely novel entities. It's not obvious that the same cognitive tools apply cleanly.

More fundamentally, keeping an open mind is a disposition, and dispositions are hard to train deliberately. The literature on changing thinking styles is genuinely mixed. Some interventions that improve reasoning under laboratory conditions don't transfer to the scenarios where it actually matters - the ones where you're emotionally invested, professionally identified with a position, or just tired.

I also can't claim that engagement with AI thought experiments produces better practical judgment about AI systems. The connection is plausible. The evidence is weak.

FAQ

Isn't skepticism about AI consciousness the scientifically responsible position?

Skepticism is responsible when it's the conclusion of scrutiny, not a substitute for it. The hard problem of consciousness remains unsolved; neuroscientist Christof Koch has acknowledged in multiple public exchanges that we lack a reliable theory of what generates experience in biological systems, which means we also lack a principled basis for ruling it out elsewhere.

What if I engage with a thought experiment and conclude AI definitely can't be conscious?

That conclusion, reached after following the argument carefully, is exactly what thought experiments are for. The process produces the value, not any particular outcome. A confident, reasoned rejection is infinitely more useful - to yourself and to the broader conversation - than an unconsidered one.

The question of open-mindedness in AI thought experiments connects directly to how we reason about novel entities more broadly - how humans have historically failed to update when confronting genuinely unprecedented things, and what cognitive habits help. It also connects to the practice of using AI as a thinking partner rather than a query machine, which changes what you bring to the conversation. And underneath both of those is a question worth sitting with: what do you lose when you decide the question isn't worth asking?