All I can tell you is that my server was hammered by a couple of IP addresses. When I did a lookup they were ALL openai and they just would not stop. Until I added protection that is. Then they got bogged down and eventually stopped.
Ive heard talk that their engineers said they dont need to abide by robots.txt since they are not an indexer like google. Which is BS.
Im not sure. I agree with you.
All I can tell you is that my server was hammered by a couple of IP addresses. When I did a lookup they were ALL openai and they just would not stop. Until I added protection that is. Then they got bogged down and eventually stopped.
Ive heard talk that their engineers said they dont need to abide by robots.txt since they are not an indexer like google. Which is BS.