This doesn't surprise me, but it should be clear this has large implications for even non misaligned models and data. What I mean is, presumably this also transfers to other biases even if more subtle or socially acceptable. If GPT 4o prefers Obama over Trump or Germany over France, all of its other output presumably will carry that bias. And a ton of information on the internet has been generated with it, and all of the other LLM models over the last few years. So we're creating a kind of perpetual stew where the output of these models is mixing with all out our chatter, and getting fed back into them. Maybe that's a good thing, maybe its kind of a mean reversion as their biases blend into a homogeneous goop. Grok's tantrum the other week might have produced poisoned data that has crazy biases in it. But in another few months that deviation from the mean will get smoothed out with new data. These are vector spaces of hundreds of thousands of dimensions per layer, it shouldn't be too surprising that biases in some areas impact the whole structure and can be transferred or reconstructed.