On my last trip to DC my youngest was 6 and my friend had a 2 year old. The 6 year old (not a confident swimmer by any means) loved the whole thing. They let him feed the dolphin rather than swimming (as he was nervous about doing this) The 2 year old was very happy just playing in the sand and splashing in the pools, as well as seeing the fish & birds. The trainers were very obliging when my friend asked at the end of the day if the 2 year old could see a dolphin up close. They brought one right up to the shallows and got the photographer to take photo's of us all as a group with the dolphin.
With young children it's a good idea to rent a Cabana for the day, as it's nice to be able to get out of the sun if you need to. It cost us $50 to hire one for the whole day.

I would go here with children of any age!