awful.systems
  • Communities
  • Create Post
  • heart
    Support Lemmy
  • search
    Search
  • Login
  • Sign Up
David GerardMA to TechTakesEnglish •
edit-2
20 days ago

Fighting the AI scraper bots at Pivot to AI and RationalWiki

pivot-to-ai.com

message-square
7
  • cross-posted to:
  • fuck_ai@lemmy.world
35
external-link

Fighting the AI scraper bots at Pivot to AI and RationalWiki

pivot-to-ai.com

David GerardMA to TechTakesEnglish •
edit-2
20 days ago
message-square
7
  • cross-posted to:
  • fuck_ai@lemmy.world
We’ve covered the AI scraper bots before. These just hit web pages over and over, at high speed, to scrape new training data for LLMs. They’re an absolute plague across the whole World Wide Web and…

video version

that s3kr1t method: https://www.jwz.org/blog/2025/05/user-agent-blocking/#comment-259266

  • RVGamer06
    link
    fedilink
    English
    0•10 days ago

    https://anubis.techaro.lol/

TechTakes

!techtakes@awful.systems

remote_follow_modal_title

Create a post
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !techtakes@awful.systems

Big brain tech dude got yet another clueless take over at HackerNews etc? Here’s the place to vent. Orange site, VC foolishness, all welcome.

This is not debate club. Unless it’s amusing debate.

For actually-good tech, you want our NotAwfulTech community

  • 198 users / day
  • 1.58K users / week
  • 2.14K users / month
  • 6.2K users / 6 months
  • 1.98K subscribers
  • 925 Posts
  • 26.4K Comments
  • Modlog
  • mods:
  • David Gerard
  • BE: 0.19.3
  • Modlog
  • Instances
  • Docs
  • Code
  • join-lemmy.org