Google (used to) scrapes the specific details authorized by robots.txt and uses it to make your content visible.
OpenAI scrapes everything it can technically see, ignoring robots.txt and feeds i to a black box and regurgitates it claiming it’s something new, that it deserves to be paid for.
Google (used to) scrapes the specific details authorized by robots.txt and uses it to make your content visible.
OpenAI scrapes everything it can technically see, ignoring robots.txt and feeds i to a black box and regurgitates it claiming it’s something new, that it deserves to be paid for.
Quite different actually.
So if OpenAI complies with Robots.txt files then there’s no issue right?
Because then they’re identical. Open AI spent a bunch of money building a powerful system they feed those results to, as did Google.