For one thing, they don't float like ducks...they sink, which is good for them as they fish underwater and have to be able to stay down, and a duck's buoyancy would work against them.
Unlike most other water birds, they don't grab fish with their beak, they spear them.
Seems to be a thin line between reptile and bird.