Most lenses only care about the angular light distribution, essentially averaging over the lens surface. So a ding or crack would just add a tiny portion of grey or black to the entire image

Edit: This is for cracking the front protective glass of a camera lens. If an actual optical lens splits in two it will be misaligned and bad things will happen.