You should not confuse religion with ethnicity. Catholic Encyclopedia is very clear:
Excluding some 30,000 Albanians living in the south-east, the Jews who emigrated in earlier times from Spain, a few Osmanli Turks, the merchants, officials. and Austrian troops, the rest of the population (about 98 per cent) belong to the southern Slavonic people, the Serbs.
http://www.newadvent.org/cathen/02694a.htm
Andric was born in Roman-Catholic family but he considered himself a Serb for the most of his adult life. His work and his last will are very clear.
In the early 20th century, both the Serbs and the Croats of Bosnia claimed the Muslims as part of their ethnic group--eventually the Tito regime created a new ethnic category for the Bosnian Muslims. In earlier times when the peasants were illiterate and nationalism had not spread to that area, a lot of Bosnians may not have identified themselves as Serbs or Croats but only by religion.
I knew that in his last 30 years Andric did not identify with the Bosnian Croats but had not heard that he called himself a Serb...there were a lot of people in Yugoslavia who avoided picking an ethnic group and just called themselves "Yugoslavs."