Historical accident. One could ask the same question about Belgium, Austria, Iraq, Jordan, Canada, most of the Central American and South American republics, and most of the countries in Africa.
The Bosnian Muslims and Bosnian Catholics don't want to be part of Serbia. The Bosnian Orthodox and the Bosnian Muslims don't want to be part of Croatia. The Bosnian Orthodox and Bosnian Catholics, for the most part, would rather be part of Serbia or Croatia, respectively, than under the Bosnian Muslims. The previous set-up was no decent solution. The current set-up is no decent solution. There is no good solution.
You should not confuse religion with ethnicity. Catholic Encyclopedia is very clear:
Excluding some 30,000 Albanians living in the south-east, the Jews who emigrated in earlier times from Spain, a few Osmanli Turks, the merchants, officials. and Austrian troops, the rest of the population (about 98 per cent) belong to the southern Slavonic people, the Serbs.
http://www.newadvent.org/cathen/02694a.htm
Andric was born in Roman-Catholic family but he considered himself a Serb for the most of his adult life. His work and his last will are very clear.