There has been a off-handed remark I heard on The Delta Flyers that Colm must have had so much time off to film other things in his contract and Terry saying that is why they never got to do anything.

EDIT: Basically any time Colm isn’t in an episode of DS9, he was off filming other things.