You guys..these are all wonderfully helpful suggestions :P...but,seriously...is there anything I can use to get rid of them?
we have those too. i have not found anything that acts as an effective repellant however an old badminton racket works especially well as a bee bat. if you hit them just right they wind up in little pieces. those sorts of bees rarely sting. good luck.
Right after dark, spray Raid wasp killer directly in the holes.