I have this little pet theory:

What if the memory is simply overwritten? A toddler, having not much experienced much in life may remember some stuff but as soon as new memories are formed the “old” memories are overwritten…

I mean, it at least would explain why my 2 year old has recently run an eerily convincing musket drill and bayonett charge with a stick while i am pretty sure he never has seen something like that on TV…