It says Noah was perfect. It doesn’t expressly say his wife or sons or their wives were though, as you may be pointing out, it is implied. I’m not hostile to the view that one or two the sons’ wives may have been carriers (though some of the other things some people come up with taking this idea out can get a little out there). Giants pop up in the lineage of Canaan and less so in some of his brothers. A few may have popped up in Japheth’s descendants. None are mention in Shem’s.
One ‘popped up’ in front of David and ranted,
“Am I a dog, that you come at me with sticks?”