I suspect it is model collapse. Too much synthetic data to steer it into useful activities. I am struggling with this on a personal project at the moment where I am trying to rip out all of the LLM-ese. I have a 31 point list of faults and an entire rubric for the clanker to follow in an attempt to mitigate the LLM-ese register. It still fails on the regular.
I suspect it is model collapse. Too much synthetic data to steer it into useful activities. I am struggling with this on a personal project at the moment where I am trying to rip out all of the LLM-ese. I have a 31 point list of faults and an entire rubric for the clanker to follow in an attempt to mitigate the LLM-ese register. It still fails on the regular.
Please share. It drives me insane.