It is easy to confuse the size of the area with the size of population. It is true that Serbs inhabited the most of area of Bosnia, but Muslims were more numerous. The reason is that Serbs were more rural and most of Muslims lived in cities. Same way Vermont, New Hampshire and Main make most of the area of New England while Massachusetts, Connecticut and Rhode Island make most of population. It does not meant that northern part was aggressive and captured the land from the south.
And ten villages around the town do not make a "sea".