My thoughts about this topic are here. It includes an analogy and me falling for Poe’s Law.

To keep it short: it’s “brittle” because it is not reasoning. Reasoning allows you to consistently transform sane input into correct output; and that is not what we see with those large token models dammit.

inb4 “but people brainfart!” - yes, when they don’t reason properly. That doesn’t change jack shit, unless you’re going to claim the large token model is extra lazy today. Mmmh.

They label it “reasoning” mostly because suckers easily fall for this sort of trap, where a mislabelled thing is confused with the real deal.