Pro@programming.devM to Technology@programming.devEnglish · edit-213 days agoCodeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.i.imgur.comimagemessage-square97linkfedilinkarrow-up150arrow-down10file-textcross-posted to: technology@lemmy.world
arrow-up150arrow-down1imageCodeberg: army of AI crawlers are extremely slowing us; AI crawlers learned how to solve the Anubis challenges.i.imgur.comPro@programming.devM to Technology@programming.devEnglish · edit-213 days agomessage-square97linkfedilinkfile-textcross-posted to: technology@lemmy.world
minus-squareredjard@lemmy.dbzer0.comlinkfedilinkEnglisharrow-up1·13 days agoCrawl-delay is just that, a simple directive to add to robots.txt to set the maximum crawl frequency. It used to be widely followed by all but the worst crawlers …
Crawl-delayis just that, a simple directive to add to robots.txt to set the maximum crawl frequency. It used to be widely followed by all but the worst crawlers …