OK, yes, but that’s just semantics.

Technically pretraining and finetuning can be very similar under the hood, with the main difference being the dataset and parameters. But “training” is sometimes used interchangeably with finetuning in the hobbyist ML community.

And there’s a blurry middle ground. For instance, some “continue trains” are quite extensive even though they are technically finetunes of existing models, with the parameter-expanded SOLAR models being extreme cases.

It’s possibly talking about its system prompt.

You are right, this is technically not its internal system, though practically something that’s hidden from end users.

It doesn’t though. Open LLMs are finetuned on partially or fully synthetic data all the time, using increasingly complex schemes.

Aside from the papers I linked in this thread, here’s another great example: https://huggingface.co/deepcogito/cogito-v1-preview-qwen-32B

Generally it’s not though. The vast majority of “swayable” X users are getting a biased chatbot, “based” leaks like this meme are the exception.

No I was thinking fully synthetic data actually.

So the prompt to make it would start with short conversations or initial questions and be like “steer this conversation toward whine genocide in South Africa”

Then have grok talk with itself, generate the queries and responses for a few rounds.

Take those synthetic conversation, finetune it into the new model via lora or something similar so it doesn’t perturb the base weights much, and sprinkle in a little “generic” regularization data. Wala, you have biased the model with no system prompt.

…Come to think of it, maybe that’s what X is doing? Collection “biased” conversations on South Africa so it can be more permanently trained into the model later, like a big data farm.

On a big scale? Yeah, sure. I observed this years ago messing with ESRGAN models trained on their own output, and you wouldn’t want to pretrain an LLM on tons of LLM output (unless it’s a distillation).

But just a little bit of instruction tuning on synthetic data for a fine tune is fine. This is literally how Deepseek was made: https://arxiv.org/abs/2402.03300

Also, some big strides are being made in the fully synthetic data realm: https://www.arxiv.org/pdf/2505.03335

Is it just stuffed in the system prompt? Should be easy to find out… That’s also hilariously stupid.

X could bias it ‘properly’ by training it in with some synthetic data, generated by Grok itself. Hell, I know how to do that. It generally wouldn’t comment on that type of bias, and also function better on other topics… but screw doing anything competently, right? Even if it’s a shitty, obvious lie, I guess X users will still eat it up.

This planet is so screwed.

I just switched from Android to iOS, and while I have many complaints, I’m pleasantly surprised by how “walled off” the apps mostly are. Unlike Android, they have to comply to function for the general public.

It feels a lot more like tier two, where it isn’t like a spyware implant but your banking app or whatever will still function. And yes I know it’s far from good, just talking degrees here…