minus-squarePlantPowerPhysicist@discuss.tchncs.detoOpen Source@lemmy.ml•The Open-Source Software Saving the Internet From AI Bot Scraperslinkfedilinkarrow-up32·6 days agothe scrapers ignore robots.txt. It doesn’t really ban them - it just asks them not to access things, but they are programmed by assholes. linkfedilink
the scrapers ignore robots.txt. It doesn’t really ban them - it just asks them not to access things, but they are programmed by assholes.