It’s impossible for an ISP to guarantee speeds though, because it’s not just their connection that’s being used.

Do they use this to weasel out of reasonable expectations of connection quality? Yes, absolutely. But they also can’t do anything about the speed of the server you’re downloading from.