Of course it makes a big difference. Those that would dispute this just don't know what they are talking about.
Then why do you think it is that the horse racing "market" had taken this into account by shrinking the size of jockeys, yet IRL racing has not? Is it just a market imperfection that is only being addressed now? If it's such a big difference, then I am surprised that IRL drivers haven't become small over the years.