Only in Donnie Yen movies. In real life Yip Man was a good fighter in his prime, but not nearly as good as the movies portray, and he never fought anyone with Chuck’s skill level, or Chuck’s size. He wasn’t nearly as athletic as Bruce, who could fight as well as Yip’s best students (as demonstrated when he sparred with Wong Shun Leung during a visit to Hong Kong), and Chuck could hold his own against Bruce, though Bruce was better.
Athleticism, size and weight are of little consequence in a beimo or real fight - which, as far as I know, Chuckie never engaged.
Ip Man is said to have easily defeated the teacher credited with actually training BL who was only, as you say, marginally better than Chuckie.
In the real world, Chuck Norris was a middleweight full contact karate champion, at around 185 lbs.
Bruce Lee was an actor, who never participated in full contact competitions, who weighed about 135 lbs.