The answer is in the wording: “why was I denying”. Always the rejection first before evaluation. To say no is the default position. In that sense its not a question of majorities. It is his own personal search as to why he was saying no to the Rosary. The mention of the multitudes who believe and use the rosary is not an appeal to authority by way of majority but the inner struggle to define the reasoning for his own personal rejection of so many.
In other words, voting on the Rosary, millions use it, I reject it, but after reflection, it is millions and I am only one so I'm going to use it too. So it isn't a question of authority or efficacy, it is a variation of 'why everyone else is doing it', why not.