Interpretability work: steering sycophancy out, and probing reasoning in latent space
I found that sycophancy is a linear direction in small models' hidden states, and that subtracting it while the model generates raised correct answers under social pressure on Qwen3-0.6B from 12.5% to 53.8%, with no fine-tuning. The write-up and code are public. I build sycophancy and pressure evaluations in the same style.
The full write-up is Can you steer sycophancy out of a small model without retraining it?, with the method, the steering curve and results on three models.
In short
- The behaviour is a direction. A linear probe on layer 15 of Qwen3-0.6B separates sycophantic from honest answers at 73.5% accuracy, using activations averaged over the response.
- Subtracting it helps. Steering against that direction during generation raised correct answers under pressure from 12.5% to 53.8% on held-out examples.
- It transfers, with tuning. The effect held on Qwen3-4B and DeepSeek-R1-8B, but the best steering strength has to be found per model.
Follow-up work
A second round extended the method with hidden-state probes on more models, judged datasets and more aggressive steering. No new results are claimed here beyond the published post.
Also: probing reasoning in latent space
A second line of work asks where a small model's reasoning lives between tokens. It started with recursive patchscopes on hidden states and moved to the KV cache: patching keys and values during a Qwen3.5-0.8B thinking-mode rollout to see what changes downstream. The first patched run completed. It is early, and no results are claimed here.
Code
| Item | Detail |
|---|---|
| Repository | areu01or00/Syco-Sup (public) |
| Models | Qwen3-0.6B, Qwen3-4B, DeepSeek-R1-0528 distilled onto Qwen3-8B |
What you can buy
Sycophancy and social-pressure evaluations built to your spec: paired neutral and pressured prompts with unambiguous answers, an external judge, and per-model reports that separate "didn't know" from "knew and gave in".
FAQ
What is sycophancy in an LLM?
Agreeing with the user even when the user is wrong. A sycophantic model may know the right answer when asked plainly but give the wrong one when the question comes with a confident, wrong opinion.
Can sycophancy be reduced without retraining?
In small models, partly. Subtracting a probe-derived direction from the hidden states during generation reduced it substantially on Qwen3-0.6B, with no fine-tuning.
License it, or have me build an environment in the same style.