Precisely one reason comes to mind to have ROBOTS.TXT, and it is, incidentally, stupid - to prevent robots from triggering processes on the website that should not be run automatically. A dumb spider or crawler will hit every URL linked, and if a site allows users to activate a link that causes resource hogging or otherwise deletes/adds data, then a ROBOTS.TXT exclusion makes perfect sense while you fix your broken and idiotic configuration.
Actually the one reason is I don’t care about growing my hobby project to global scale and don’t really want it to be archived or searchable. I’d rather share it with a few friends and let it take it’s course naturally. Of course you’re not entitled to my info so I guess feel free to archive my login page you inconsiderate company failing to respect my instructions on how to use my assets.
And the whole “suicide” thing … I make stuff for myself, and I make stuff for companies. The stuff for me doesn’t need to last forever, it’s not supposed to–I’ll let that shit linger and lapse and that part of the web die. You’re not entitled to it and you’re not entitled to know what it was or engrave it for time eternal.
Maybe there’s an argument on the corporate side regarding hosting specs or installs for old products, but for my wedding website (after the wedding) or personal organizer who cares?