Where do you get the real data, though? They just scrap data from websites
Great question… Do they “just” scrape data from websites?
https://www.theatlantic.com/technology/archive/2025/03/libgen-meta-openai/682093/
Keeping it clean would require hiring people to scrub contamination from the data sets.
That’s exactly right.
The point I was trying to raise that wasn’t semantics was that if the majority of the full training data were synthetic, it could lead to model collapse.
But luckily (or not?) a small amount of finetuning can be very effective in correcting the range of responses.
Open LLMs are finetuned on partially or fully synthetic data all the time
That’s what I was suggesting.
You explained to me you weren’t talking about “finetuning”, but training on completely synthetic data.
(Fine-tuning happens after the LLM has already been trained)
Oh, yeah then I agree with above commenter. This would collapse the model.
You’re not training an LLM on text generated by an LLM. You’re training it on 98% real data, and intentionally biasing it by sprinkling in the fake data intermittently.
People aren’t interested in “learning about LLMs”, especially people like artists.
They’re interested in telling Elon Musk to “fuck off”, and when Grok says something bad about Elon it’s very cathartic for them.
They might know it’s feeding their own thoughts back to them, but they don’t care. To people who aren’t in the know, this box Elon is promoting as “objective truth box” is criticizing Elon. That’s a very powerful narrative in a world where he’s taking over the world.
It’s hard to disagree. Elon can go fuck himself. What’s more important to the average person, stopping Elon or understanding the nitty gritty of machine learning?
When artists say AI is stealing, they’re not interested in an explanation about how “its really not”. And if you tried to, they’d feel you’re missing the forest for the trees because their problem with AI isn’t metaphysical philosophy, it’s that it’s hurting their job opportunities.