Understanding the inner thoughts of AI

Neel Nanda frames interpretability as reverse-engineering systems that are trained rather than explicitly designed, analogous to biology studying evolved organisms. His core argument is pragmatic: full understanding is unlikely, but partial understanding can still materially improve safety, debugging, monitoring, and alignment evaluation. He emphasizes that some simple methods already work surprisingly well, especially reading chain-of-thought-like "scratchpads," probes, and sparse autoencoders, while warning that none is a silver bullet and future, more capable models may become harder to inspect honestly.

Key insights

1. Interpretability exists because modern AI systems are grown, not designed: Nanda argues neural networks are produced through repeated training nudges over vast data, not by specifying their internal mechanisms in advance. That makes interpretability less like reading source code and more like biology or neuroscience: trying to infer the functions and structures that emerged from optimization.

   Why it matters: This explains why black-box behavior alone is insufficient for confidence. If capabilities emerge from training rather than design, safety and reliability require reverse engineering, not just external testing.

2. The field is shifting from dreams of full understanding toward pragmatic usefulness: He describes an earlier mechanistic-interpretability ambition to understand models as completely as possible, partly motivated by results showing identifiable neurons or components tied to concepts. His current stance is more operational: use whatever methods work to answer concrete questions about model behavior, even if understanding remains incomplete.

   Why it matters: This is a strategic signal about where serious interpretability work may produce value soon: targeted auditing, monitoring, and debugging rather than a complete theory of model cognition.

3. Chain of thought is best treated as a scratchpad, not a transparent window into reasoning: Nanda says reasoning traces are often useful and among the best current tools, especially for investigating failures, but they are not guaranteed to faithfully expose all internal computation. The scratchpad analogy captures the limit: some work can happen off-page, some written steps may be irrelevant, and future stronger models may deliberately control what they reveal.

   Why it matters: Decision-makers should not over-rely on visible reasoning as proof of internal alignment or honesty. It is a high-value signal, but not a complete or future-proof one.

4. Simple interpretability tools can reveal real internal structure: He highlights probes and steering as surprisingly effective. In the Othello example, a model trained only on move sequences still internally represented board state, and probes could recover that representation. His broader lesson is that straightforward methods often beat elegant but impractical theory when the goal is actionable understanding.

   Why it matters: This suggests interpretability is not purely speculative. Even modest tools can recover latent variables the model was never explicitly asked to expose, which is useful for auditing and control.

5. Sparse autoencoders matter because they can surface concepts researchers did not think to ask about: Unlike probes, which test for a chosen feature, sparse autoencoders aim to discover many latent concepts automatically. In the hallucination example, they surfaced concepts corresponding to whether the model recognized an entity; manipulating those concepts changed whether the model answered or fabricated.

   Why it matters: The value is not just explanation but discovery. For safety, the important internal variable may be something humans would not have pre-specified, so methods that surface unexpected features are strategically important.

6. Interpretability is especially useful for alignment auditing because behavior is ambiguous: Nanda argues that evaluating alignment from outputs alone is hard: an apparently aligned model may be faking, and an apparently misaligned one may be roleplaying or misunderstanding instructions. He points to auditing techniques, including sparse autoencoders and a black-box 'prefill attack,' as ways teams can uncover hidden objectives that pure API access may miss.

   Why it matters: This reframes interpretability as an evidence-quality tool. The main benefit is not just spotting bad behavior, but distinguishing dangerous hidden objectives from benign confounders before deployment.

Strategic implications

• Interpretability is likely to be most valuable near deployment gates: auditing new models, investigating suspicious behavior, and improving confidence in alignment evaluations rather than certifying complete safety.
• Safety strategy should assume defense in depth. Nanda explicitly rejects interpretability as a standalone solution and positions it as one layer among multiple imperfect safeguards.
• Organizations that only evaluate models through black-box APIs may systematically miss hidden objectives that deeper-access teams can detect, implying a governance and infrastructure advantage for labs with internal interpretability capability.
• A practical roadmap is emerging: build cheap, effective monitors for deception, misuse, and unwanted behavior, and integrate them into routine model oversight rather than treating interpretability as a pure research exercise.

Signals to watch

• Whether future models remain forced to use readable 'scratchpads' for difficult reasoning, or become better at hiding or compressing the real reasoning process.
• Progress on sparse autoencoders and related methods from explanatory tools toward reliable operational monitors for hallucination, deception, or hidden goals.
• Whether internal auditing methods consistently outperform black-box evaluation on alignment-relevant tasks, especially before release.
• Evidence that interpretable internal variables can be causally edited in robust ways, not just observed, to reduce failure modes such as hallucination or deceptive behavior.

Caveats

• The transcript includes omitted middle sections, so some examples and supporting detail are missing; only claims clearly present in the provided text are included.
• This is an interview-format discussion, not a technical paper. Several claims are framed as judgments, expectations, or research directions rather than established consensus.
• Some examples are described at a high level without methodological detail or quantitative results in the transcript, so their strength should not be overstated.