OpenAI publishes early sparse-autoencoder work extracting concepts from GPT-4

OpenAI published its own sparse-autoencoder interpretability work, training an autoencoder with 16 million features on internal activations from GPT-4 and reporting that many of the resulting features corresponded to human-interpretable concepts. The technique — training a smaller model to reconstruct a network’s internal activations from a sparse set of active features, on the premise that each feature captures a single, nameable concept rather than a tangle of superimposed ones — followed the same general approach Anthropic had published weeks earlier at a larger scale on its Claude 3 Sonnet model.

OpenAI reported that its new training methods scaled smoothly and predictably to tens of millions of features and outperformed prior techniques on that basis, but was explicit about the method’s cost: passing GPT-4’s activations through the autoencoder degraded performance to roughly what a model trained with ten times less compute would achieve, and the team estimated that fully mapping the concepts inside a frontier model might require billions or trillions of features — a scale their improved methods would still struggle to reach.

The near-simultaneous publication of comparable work from OpenAI and Anthropic within weeks of each other marked interpretability research, previously a niche academic pursuit, as newly competitive between the largest labs — evidence that understanding what a model represents internally had become part of the same race as building bigger ones.