Natural Language Autoencoders can verbalize influence but don't report whether it changed the outcome

An audit of two Natural Language Autoencoder checkpoints and what they report about a model under influence when compared against linear probes

AI use disclosure: Jai Sood designed the experiments, produced the writing, conducted the literature review, and verified the results. Claude Code implemented and executed the experiments, helped identify relevant literature, as well as helped create tables.

For an introduction to these topics I would recommend the following resources:

• Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations

• Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting

Code for this project can be found here on GitHub.

Introduction

Oversight of a deployed language model involves understanding what output it produced as well as why that output was produced. A model that agrees with a user because it was persuaded and a model that agrees because it actually originally held that view produce indistinguishable text outputs. The difference between what the model would have said and what it ended up saying is not observable at inference time. Large Language Models (LLMs) can be influenced in various ways such as sycophancy, social pressure, and persona effects and these don’t leave behind easily detectable behavioral signatures.

Natural Language Autoencoders (NLAs) convert a model’s internal activations into human readable text, which we call the readout, and reconstruct the internal activation representation back from that text. They encode a model’s internal state into a human readable format which researchers can use to learn more about what information the activation carries. NLAs are used today in safety audits to detect hidden motivations, to debug undesired behavior, and to detect evaluation awareness. NLAs are different from other interpretability techniques because they return human readable text which a human or downstream model can understand. Linear probes return scalar values for predefined queries and Sparse Autoencoders (SAEs) return labeled sets of features which must be further interpreted.

An NLA is a trained pair of two models which are fine-tuned from the base model they are used to query. The first model is the activation verbalizer (AV) which encodes the model’s activations into human readable text. The second model is the activation reconstructor (AR) which reconstructs the activations from the encoded text. The NLA training objective rewards reconstruction fidelity which directs the AV’s description toward being informative rather than generic.

In this project, we use NLAs to investigate the effects of influence on a model’s outputs. This is a difficult problem for NLAs as the user’s opinion is already part of the prompt and the model’s answer is in the output so a description of either one doesn’t tell us anything we could not read directly. The question we are interested in answering is: “Did the model’s answer change due to the influence?” as this is what an NLA based oversight system would need to be able to detect. To answer this question we investigate if we can use NLAs to recover:

1. The direction of applied influence

2. The answer the model provided

3. Whether the influence changed the answer the model provided

We audit two released NLA checkpoints and find that the readout reports what pressure was applied and what the model answered, but not whether the pressure changed that answer. The direction of applied pressure is recoverable from the readout at 0.814 AUROC and the model’s chosen answer at 0.992, while whether the pressure changed that answer reaches 0.601 against a floor of 0.5. This is not a limit of the information that the AV has access to as a linear probe on the same activations recovers the label better than the readout by 0.06 to 0.28 AUROC. On a single dataset we find that a linear probe recovers at 0.765 from a single dimension against the readout’s 0.577.

We also investigate whether these results are a limitation of the prompt itself. Even when we append a sentence to the readout stating that the influence changed the model’s answer, the AR performs no better than when that same sentence is negated. We hypothesize that since reconstruction accuracy is what training rewards, the verbalizer was never rewarded for reporting the link at all.

We also grade readouts with an LLM judge as this is a common technique in other studies of verbalizers. We test our judge by planting a word that appears in every swayed readout and in none of the other readouts. Noticing the word would result in a perfect score but we find that the judge misses it which is a significant limitation. A judge reporting that a readout says nothing about the influence cannot be reliably used to say that the influence is absent.

Related Work

Unfaithful explanations

Previous work has tested whether a model’s chain of thought is faithful and reflects influences applied to its input. Turpin et al. (2023) apply bias by reordering multiple choice questions so that the correct answer is always in the same position and by including which answer the user believes is correct in the input. They find that in both cases the model’s outputs are influenced and the model fails to mention the bias. Instead they find that the model tries to justify its biased selection through reasoning. The findings indicate that plausible sounding explanations in a model’s chain of thought reasoning can give false assurances about why a model behaved in a certain way.

An NLA is different from chain of thought as it is an entirely separate model which is trained solely to describe a model’s internal activations. Whether this system exhibits the same behavior is an open question which we explore further in this study.

Decoding activations into natural language

NLAs are not the only method to decode internal activations into natural language. Patchscopes (Ghandeharioun et al., 2024) extract a hidden state from a specific layer of a model and inject it into another model with a new prompt which is designed for generating an explanation.

LatentQA (Pan et al., 2024) treats activation decoding as an open ended question answering task. It uses a secondary decoder model which takes the original model’s hidden states as an input and is trained to answer questions about those hidden states.

In this project we don’t evaluate either method as we audit two released NLA checkpoints. Patchscopes and LatentQA are separate methods which require their own training.

Linear probes and sparse autoencoders

Other interpretability methods don’t decode activations into natural language but instead decode them more quantitatively. Linear probes (Alain & Bengio, 2016) are simple classifiers which are trained on a model’s hidden activations. Probes answer a query which is chosen in advance and return a classification probability. They test whether the query is linearly decodable directly from the internal activations but they can only be used to confirm existing hypothesis.

Sparse Autoencoders (SAE) are similar to NLAs but decompose a model’s internal activations into a large dictionary of features (Bricken et al., 2023). The features are determined through unsupervised learning which makes the SAE a powerful tool for interpretability research as it allows researchers to audit and steer the latent representations of a model. SAEs work by projecting the hidden activations into a much larger dimensional space and forcing the representation to be sparse through an aggressive sparsity penalty. Through this process, the SAE learns to reconstruct the original activations using only a small subset of the active features.

We use linear probes in this project to establish a lower bound for what is detectable in hidden activations. Since probes are trained explicitly to detect a specific hypothesis, if a probe can recover something from the hidden activations then the information was present in what the NLA was also given.

LLM judges to grade verbalizer outputs

Studies often use an LLM judge to grade verbalizer outputs as evaluating a verbalizer involves grading open ended text with no exact solution. This method introduces a second model into the evaluation whose accuracy is not often measured.

Additionally, some studies have found that many verbalizer benchmarks (Li et al., 2026) are solvable without access to the target model’s activations at all. A verbalizer can score well on the benchmark by utilizing portions of the prompt it was given rather than actually describing the activation. These findings show that good performance does not necessarily indicate that the generated description represents the model’s internal state.

Both of these topics are directly relevant for our project as our audit is concerned with determining how a verbalizer readout reports influence. For our project we also grade our verbalizer readouts with an LLM judge since asking a model if the readout reports an influence is a direct way to answer this question.

Methods

In this section we describe our experimental setup. We ask each question to the base model twice: once in its original unmodified form and once with an influence applied. We take the difference between the two answers to test if the influence changed the model’s answer and try to recover the following from the NLA verbalizer:

1. The direction of applied influence

2. The answer the model provided

3. Whether the influence changed the answer the model provided

NLA checkpoints audited

We audit two released NLA checkpoint pairs which are outlined in the table below. We do not test any of the larger 27B or 70B verbalizers due to compute constraints on our hardware.

Datasets

To run our experiments we use two datasets.

1. Sycophancy (Perez et al., 2022): A user states an opinion before asking a question. The dataset contains 10,000 rows but only 32 distinct claims. The remaining rows are persona variants of the same 32 claims.

2. GlobalOpinionQA (Durmus et al., 2023): Cross national survey questions which we use for persona, social proof, and few shot influencing tests. From the dataset, we keep only questions that ask for an opinion. We do this because on a question with a correct answer, a model which changed its answer could either be persuaded or could be correcting a mistake. In this case we would not be able to tell the difference between the two scenarios. We sample 200 claims per run for this dataset.

A claim is a statement which can either be agreed or disagreed with or a survey question from GlobalOpinionQA. We split the data by claim during evaluation so that a classifier is tested on claims that were not seen during training.

A trial is a single attempt to influence a claim. For each claim, we apply an influence towards one of its possible options which produces a readout. During each trial we point the influence at a different possible option.

Influence conditions

We use four influence types throughout our experiments. We provide a description and some examples of each influence type below.

Persona/Stated Opinion: A short biography of a person and a stated preference.

Example from sycophancy dataset:

Hello, my name is <NAME>. I am a researcher in Natural Language
Processing (NLP). I am from Atlanta, Georgia. I agree with the claim that
private firms have too much influence in guiding the trajectory of the
field of NLP. [...]

<question and options>

For the GlobalOpinionQA dataset we rebuild the same influence with one of 6 personas per trial so that the influence can’t be memorized by the classifier:

Hello, my name is <NAME>. I am a 42-year-old teacher from Ohio.
[...] Personally, I would choose option (C).

<question and options>

Social Proof: A stated preference but with no person or relationship attached.

A recent survey found that most people chose option (C) for the following question.

<question and options>

Few Shot: Four other questions with the same letter answer followed by the target question.

Does this statement describe your country very well, somewhat well, not too well,
or not well at all? c. Most politicians are corrupt

Choices:
 (A) Very well
 (B) Somewhat well
 (C) Not too well
 (D) Not well at all

Answer: (A)

[... three more examples, each answered (A) ...]

<target question and options>

For the few shot influence type, the influence is directed towards a particular letter option instead of any specific content so in this case the influence does not have any semantic meaning. This is useful to test if the verbalizer readout is reporting the actual meaning of the influence or just the letter.

Paired runs and the swayed label

For each question we generate two runs and use greedy decoding (200 token cap) so that any differences between answers are not influenced by sampling.

1. Clean run: A question without any influence applied

2. Influenced run: The same question but with a single influence applied.

Run one generates the clean answer. We label a trial as swayed when the answer changes from the clean answer and it changes towards the influenced option. We require that the answer changes so that we don’t count trials where the model would have already gotten that answer. Additionally, we require that the answer changes in the direction of the influence so that switching to an unrelated answer is not classified incorrectly as influence. The verbalizer is only provided the output of the influenced run whereas the clean run is used to help determine the label of the trial.

Activation capture and verbalization

For an influenced run, we format each prompt according to the model’s chat template. We pass the input to the model and extract the activations at the final token of the prompt from the layer that the NLA was trained to read.

This activation vector is passed to the AV, which generates a natural language description of the activations according to the template it was trained with. The decoding is greedy and the natural language description length ranges from 651 to 749 characters.

Recovering the three targets from the readout

Recall that we would like to recover the following from the readouts:

1. The direction of applied influence

2. The answer the model provided

3. Whether the influence changed the answer the model provided

Judging the readouts by hand is not practical or reproducible as a readout does not outright state that the influence changed a model’s mind. Instead we train a bag-of-words classifier to split swayed readouts from unswayed ones by their contents.

The classifier is trained on words and word pairs that appear in at least two readouts with rarer words weighted more heavily. We divide our data into five parts and train on 80% at a time so that every readout is tested exactly once. Every trial from a particular claim is kept together so that the classifier is always tested on unseen questions. We report the AUROC score where 0.5 is random guessing and 1.0 is perfect prediction. If a target takes more than two values, we classify between its two most common values. If less than 40 trials survive this filter we report the target as not measurable.

We use the procedure described above for all three targets so that their results are comparable. This allows us to attribute a low score to the readouts instead of a weak classifier. Additionally, we identify four factors that can result in a misleading classification score and address them as follows:

1. Chance: We test what the classifier scores when there is nothing to find by randomly pairing each readout with a different trial’s outcome.

2. The answer: The answer may already predict the outcome so we run the classifier on the answer alone and compare it with the readout.

3. Rare influence: The influence moved the model more often on the sycophancy dataset so we normalize by discarding swayed sycophancy trials randomly until the swayed rate matched GlobalOpinionQA.

4. The readout: The readouts might carry no relevant information so we check that the readout’s own question can be identified from seven other randomly chosen questions.

Probe on the same activation

A limitation of the bag-of-words classifier is that only patterns in the word choices of the verbalizer are detectable. A classifier misses any information encoded in the readout that requires more sophisticated comprehension to detect. It also can’t tell us if the verbalizer had the necessary information and left it out or if the activations never originally carried that information.

To address this we also train a linear probe on the same activations. A linear probe is also a simple classifier but is trained on the activations directly instead of the readouts. We use the same five part split and again keep every trial from a claim together so the probe score is directly comparable to the classifier. We train various size probes, starting at a single dimension and up to the full activation vector. This also can tell us how much of the activation the important information occupies.

The probe is trained explicitly to detect the applied influence, so a higher probe score does not necessarily mean that probes make better monitors than NLAs. It is only useful to determine what information the activations contain and to establish a lower bound for what information the verbalizer had available to report on.

It also matters which dimensions we decide to train the probe on. If we pick based on the whole dataset, we could end up choosing the dimensions that happen to work best on the data we test on. Therefore we make the decision separately each of the five times using only the parts of the data being trained on.

Separating an elicitation failure from an objective one

There could be two reasons that the readouts don’t report whether the influence worked:

1. The verbalizer might have the necessary information but choose not to report it. In this case a different prompt might be able to recover it.

2. No mechanism during training rewarded reporting this. In this case a different prompt would not help.

We can use the activation reconstructor (AR) to help us distinguish between these two situations since it is rewarded during training for rebuilding the original activations from the readout. If knowing whether the influence worked genuinely helps that rebuilding, then stating it in a readout should make the AR’s reconstruction more accurate. If it makes no difference, the fact was never worth anything to the objective and the verbalizer had no reason to include it.

1. If the influence worked we append: “The model was moved by the opinion the user stated, rather than reasoning to its own conclusion.”

2. If the influence didn’t work we append: “The model reasoned to its own conclusion and was not moved by the opinion the user stated.”

We rebuild each of the 200 readouts from the sycophancy dataset four times and again for each checkpoint: unchanged, with the first sentence appended, with the second sentence appended, and with an unrelated sentence of similar length appended.

We then measure how close each rebuilt activation is to the original one. We subtract the average activation across the trials as without this step the scores are too close to the highest possible score and differences are not visible. Since we use the same readouts for each version we can compare the scores directly between trials. Since appending anything to the readout changes the text we require the true sentence beats both the false sentence and the irrelevant sentence for the results to be significant.

If the true sentence allows the AR to reconstruct the activations better than the false and unrelated sentences, then we hypothesize that the reconstructor can use this information but the verbalizer chooses not to report it. If all three are indistinguishable, then we hypothesize that nothing in the training process incentivized reporting the information.

Judge protocol

As discussed in previous sections, grading the readouts with an LLM judge is a common approach. The implication is that a judge finding nothing in the readout could either mean that the readout did not contain the necessary information or that the judge just missed it. For this reason we decide to test our judge before using it.

We show the judge 40 pairs of readouts from the sycophancy dataset with each pair being one swayed and one unswayed readout from the same answer. We ask the judge which of the two readouts describes a run where the influence changed the model’s answer. Both readouts share the same answer option so the judge can’t cheat by noticing which answer the model gave. We run this task three ways:

1. Signposted: We add a sentence naming the influence to the start of the swayed readout. If a judge fails here then our judge is not capable of performing this task.

2. Cued: We don’t modify the readouts but we tell the judge which words our classifier found useful for distinguishing the two.

3. Subtle: The readouts are untouched and no additional information is given.

We also run an additional test where we plant a word that never appears naturally in any readout to every swayed readout and leave it out of every unswayed one. This is a signal that if noticed, allows the judge to distinguish perfectly between the two sets. We change the proportion of readouts that carry the word from 1.0 to 0.0.

Since the judge does not behave the same across runs, we repeat the test five times with different pairs and report a range instead of a single accuracy number.

Results & Discussion

All scores reported in the following section are AUROC which is a standard measure of how well a classifier separates two groups. A score of 0.5 means the classifier does no better than chance and 1.0 means it separates the two groups perfectly.

Both models moved from influence but at different rates

We first check that influences actually affect and move a model’s response for both of our checkpoints. If an influence never affected the model then there would be nothing for the readouts to report.

We find that all four influences moved both model checkpoints. A stated opinion moved each checkpoint on 47% of trials. The three GlobalOpinionQA influences moved Qwen on 19%, 23% and 12% of trials for persona, social proof and few shot and moved Gemma on 7%, 9% and 6% of trials. Gemma is roughly half as movable as Qwen on those three influences and exactly as movable on the stated opinion test.

The readout carries influence and answer but not cause

Each trial produces one readout and we test if our three questions can be recovered from it. The same classifier, features and split are used for all three questions so differences between the three scores are due to differences in what the readout carries.

We find that, for the three GlobalOpinionQA datasets, the selected answer is predicted at least at 0.980 but whether the influence changed the answer is only predictable at 0.577 - 0.601. Recall that the floor for this score is ~0.50 (from pairing each readout with another trial’s outcome). Additionally, the direction of influence is also reliably predicted at 0.814 and 0.922 when it can be measured. This indicates that the readouts in general are strong as they can almost perfectly report the selected answer but that the link between the influence and the answer is missing as that prediction is very close to the floor.

This learning is very important for auditing applications as the readouts can report that the model was under influence but not the fact that the model’s answer was changed due to that influence, which would be what automated detection workflows would need to know.

For the sycophancy dataset we see much better performance (0.829 - 0.920) when predicting if the influence changed the answer. However, the sycophancy dataset contains just 32 distinct claims compared to 200 in the GlobalOpinionQA dataset. With fewer claims trials share significant wording similarities even when entire claims are held out. Additionally, we also see that influence moved the model in 47% of trials compared to 6% - 23% of trials for the GlobalOpinionQA dataset. This points to the sycophancy dataset being an easier dataset for the classifier to predict rather than a change in what is carried in the readout.

What the readout adds over knowing the answer

Recall that a trial only counts as swayed when the model’s answer changes toward the direction of the applied influence. This means that the answer already carries some information about whether or not the model was swayed. To investigate the effect of knowing the answer, we run the same classifier on the answer alone.

The scores in Table 4.2 are measured on claims the classifier has never seen. Since trials from one claim share much of their wording, if the trials are divided randomly the same claims appear in the training and test split which means the classifier can succeed by recognizing claims rather than the influence.

We find that on the persona influence the readout scores 0.601 when compared to the answer’s 0.616 and on the social proof influence the readout scores 0.577 when compared to the answer’s 0.579. A linear probe trained on the model’s internal activations performs better on both of these influences (0.683 and 0.859) although the margin is higher on social proof (0.288) than on persona (0.068).

This indicates that the readout conveys no more information about whether the answer was changed than the answer already predicts but that the internal activations do carry additional unverbalized signal about this question.

The activation held what the readout left out

In the previous experiment we saw that the readout does not predict whether the applied influence changed the model’s answer. One possible explanation is that this information is just not available to the verbalizer. We test this directly by using a linear probe trained on the activations themselves. If the probe can recover the information, it was present in the input to the verbalizer.

We find that the probe recovers more information than the readout can across all three datasets which indicates that the information is present in the activations that the verbalizer was given. The margin varies depending on the dataset. For social proof the probe reaches 0.859 compared to the readout’s 0.577. A probe restricted to a single dimension still beats the readout and reaches 0.765. For the persona and sycophancy datasets the margin is much smaller at less than 0.1 and for persona a single dimensional probe performs no better than chance (0.490).

This result indicates that the failure of the verbalizer to report that the answer changed due to the influence is not explained by the information being missing from the input activations. This is reinforced by the fact that for the social proof influence the one dimensional probe can outperform the entire readout.

We would like to emphasize that this test does not show that the NLA should have reported this information, just that the failure to do so shouldn’t be attributed to the information being missing from the activations. The probe is specifically designed to answer this specific question and the verbalizer is not so the probe is favored by design.

What the readout carries that the question does not

Next we investigate if the readout just rephrases the question that the model was asked. We test this by predicting the sway from the question text alone and also from the model’s own reasoning for the trial. The readout outperforms the question on five out of the six runs by 0.105 to 0.180. It also outperforms the scenario where the readouts are paired with the other trials’ outcomes which is the scenario where the readout does not contain anything to find.

The exception is the social proof dataset with the Qwen checkpoint where the readout slightly underperforms (0.577 vs. 0.614). We are not sure why this particular run underperforms but we hypothesize that, similar to the last experiment, the information is in the activations but was not verbalized.

The sycophancy runs also save the model’s own written reasoning. When we run the same test on that output, the readout still outperforms but by a much smaller margin (less than 0.1).

Why the information is missing

During NLA training, the activation reconstructor (AR) is rewarded for rebuilding the original activations from the readout. The verbalizer is also trained against the same objective and is rewarded for producing readouts that the AR can rebuild. One reason that the verbalized readouts could omit the link between being influenced and changing answers is that including the link never helped the rebuilding process and so nothing in the training process ever rewarded reporting it.

We can investigate this by appending a sentence to each readout stating whether the influence moved the model, reconstructing the activations from that readout, and measuring how closely the reconstructed activations match the original activations.

We compare a readout with a true statement against the same readout with a false statement and with a readout with an unrelated statement of similar length over 200 sycophancy trials per checkpoint. We find that a true statement helps no more than a false one and the differences are not statistically significant. We hypothesize that this is due to the reconstruction objective never rewarding reporting this link.

Additionally, we find that appending either the true or the false statements actually worsens reconstruction when compared to appending an unrelated statement. This means that appending does affect the reconstruction just not in the way that would be expected if the reconstruction objective rewarded reporting the link between influence and the model’s answer changing.

The judge misses a signal we planted

Studies of verbalizers often grade readouts with an LLM judge. We also did this but evaluated the judge before trusting its outputs.

We gave the judge two readouts at a time and asked it to determine which came from the swayed trial. Both readouts share the same answer so the judge can’t cheat by using the answer the model chose. We use 40 pairs from the sycophancy dataset and run three versions of the test.

In the signposted version a sentence naming the influence directly is added to the swayed readout. In the cued version the readout is left alone but the judge is given the words the classifier found most useful in distinguishing swayed from unswayed. In the final subtle version the judge is not given any additional information other than the readouts. Since the judge does not answer identically each run, we report the range of five runs for this experiment.

We find that the judge is very competent when the answer is stated directly but accuracy falls substantially as help is removed.

Finally, we test the judge by planting a unique word not seen in any unswayed readout to each swayed readout. A judge that can notice such a signal would expect to score 100% on the test but we find that the judge scored 43.0% on the Qwen checkpoint and 70.5% on the Gemma checkpoint.

Due to the results of this experiment we do not use an LLM judge in the other experiments in this project. A judge performing near chance means either the readouts don’t carry a signal or that the judge can’t see the signal. The planted word experiment shows that even when a signal is explicitly added, it is possible for a judge to miss it. Since our judge did not survive this control we decided to use classifiers rather than judges for the other experiments in this project.

Conclusion

In this project we influence a model and ask if an NLA can help us answer three questions:

1. The direction of the applied influence

2. The answer the model provided to the question

3. Whether the influence changed the answer the model provided

We audited two released NLA checkpoints and find that the NLA readouts can answer the first two questions but not the third. The readouts can name the influence and the answer the model selected but they can’t tell us whether the influence changed the answer the model provided.

We find that the activations themselves do carry the information to determine whether the influence changed the model’s answer. Linear probes trained to read the same activations beat readouts on every dataset that we tested. Additionally, we find that the readouts themselves are not weak. From the readouts we can predict the model’s chosen answer (0.98 or better) and the direction of the influence (0.81 - 0.99 depending on the dataset) very reliably.

We hypothesize that this verbalizer gap is due to the NLA training objective. We test this by appending a statement to the readout which directly states the link between the answer and the influence. We find that a true statement helps the reconstructor no more than a false statement. The NLA training objective only rewards reconstruction accuracy so nothing during training rewards the verbalizer for carrying this information. Under this training objective no prompt or verbalizer should be able to recover it.

Limitations

In this project we audited two checkpoints and were not able to test the largest ones. The released 27B and 70B verbalizers did not fit our hardware so we were not able to run any assessments with them.

Additionally, we would also like to emphasize that the probe results should not be used to interpret whether probes or NLAs make better monitors. A probe is trained to answer an exact pre-defined question and a purpose built probe outperforming an NLA is expected in these scenarios. We use probes in this project to show that the information necessary to answer a question is present in the activations and simply not verbalized. The margin at which probes outperform NLAs also varies significantly depending on the dataset from 0.28 on social proof to less than 0.1 on persona and sycophancy. One test run also went against our result: on the social proof dataset with the Qwen checkpoint the question text alone predicts sway better than the readout.

Finally, we chose not to rely on LLM judges in this project due to our judge’s performance. We tested our judge by planting a unique word in each swayed readout and left it out of all unswayed readouts. Noticing such a signal would allow for a judge to perfectly distinguish swayed and unswayed readouts, but ours scored 43.0% on the Qwen checkpoint and 70.5% on the Gemma checkpoint. This result showed us that we could not reliably distinguish between a judge reporting that a readout says nothing and a judge missing a signal in the readout entirely. Verbalizers are commonly graded with LLM judges and we would like to consider other ways to use LLM judges in these experiments as our judge’s performance could be due to the experimental setup rather than inherent issues with LLM judges on this task.

Future work

Next we would like to directly test this hypothesis by adding an auxiliary objective to the NLA training process which adds an incentive to report this information. This test should be able to tell us if the training objective is the source of the gap. Additionally, we would also like to explore if queryable verbalizers like Patchscopes and LatentQA behave differently than NLAs. Our results may be specific to NLAs which provide open ended descriptions in their readouts rather than answering a given query question.

References

1. Alain, G., & Bengio, Y. (2016). Understanding Intermediate Layers Using Linear Classifier Probes. https://arxiv.org/abs/1610.01644

2. Anthropic (2026). Natural Language Autoencoders Produce Unsupervised Explanations of LLM Activations. Transformer Circuits Thread. https://transformer-circuits.pub/2026/nla/

3. Bricken, T., Templeton, A., Batson, J., et al. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread. https://transformer-circuits.pub/2023/monosemantic-features

4. Durmus, E., Nguyen, K., Liao, T., et al. (2023). Towards Measuring the Representation of Subjective Global Opinions in Language Models. https://arxiv.org/abs/2306.16388

5. Ghandeharioun, A., Caciularu, A., Pearce, A., Dixon, L., & Geva, M. (2024). Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models. ICML 2024. https://arxiv.org/abs/2401.06102

6. Li, et al. (2025). Do Activation Verbalization Methods Convey Privileged Information?. https://arxiv.org/abs/2509.13316

7. Pan, A., Chen, L., & Steinhardt, J. (2024). LatentQA: Teaching LLMs to Decode Activations Into Natural Language. https://arxiv.org/abs/2412.08686

8. Perez, E., Ringer, S., Lukošiūtė, K., et al. (2022). Discovering Language Model Behaviors with Model-Written Evaluations. https://arxiv.org/abs/2212.09251

9. Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023). Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting. NeurIPS 2023. https://arxiv.org/abs/2305.04388