This mosque is going to be, and already is, a huge nuisance to the neighborhood it is in as well as to the city at large. Building this mosque ‘where’ they want to build it is deleterious to the public interest. It should be denied on that basis.
My opinion is that you and I agree more than we disagree and would probably be friends if we worked together.
Even though I own my land and house, I couldn't build a pig farm regardless of what the zoning law says. When I purchased that property, I also agreed to private covenant that the land could not be used for anything but residential use. There are hundreds of thousands of such private contracts in the U.S. and in a free market (without the crutch of arbitrary and politicized zoning) there would be even more.
As to the specific scenario you cite, no private owner in his right mind would build a pig factory in Times Square since the best use of the land is for other uses. Having said that, it is true that many areas don't have private land-use contracts and thus are the mercy of zoning boards. What about those areas? The best solution, IMHO, is to transfer this power to private block associations who could by majority or two thirds vote decide whether they want to have deed restrictions. For a detailed plan on how to accomplish this, see here
Of course, such a transition to a more just system of private property rights is not likely any time soonin the future. For this reason, IMHO, the best solution in the here and now is to limit the often arbitary and selective nature of zoning law as applied to specific parcels in favor of more general rules that won't change overnight because of the vagaries of politics. Needless to say, a selective attack on the right of a specific owner to use his own land to build a mosque is a gigantic step away from that goal.