IF you would read the historical report posted at Monticello you would discover that historians took that possibility into consideration in their study. They conducted a very detailed survey of Jefferson’s plantation diary, which recorded all Jefferson male visits to his property over the years; as well as Mr. Jefferson’s comings and goings. The other Jefferson men were not on the property when Sally Hemmings conceived - only Thomas was there for each of them. Mr. Jefferson recorded all of the births of Sally Hemmings children by name. No other slave children received that distinction in his records; nor were any other slaves granted freedom on his death.
I had the same opinion as you have expressed until I read the actual historical research report.
P.S. In case you might not realize, Sally Hemmings was Jefferson’s wife half-sister. Her father and grandfather were Mrs. Jefferson’s father and grandfather. Sally Hemmings grew up with Mrs. Jefferson and was given to Mr. and Mrs. Jefferson by Mrs. Jefferson’s father when they married.