Now that depends on what they were hunting and what sort of defense. A musket could be loaded with birdshot, and used as a fowling piece, or it could be loaded with buckshot or "buck and ball", and make a good defensive weapon, although not as handy as it's short barreled cousin the blunderbuss. As you point out, rifles were expensive and slow. Slow might be a bigger problem in the "Big Woods" than "inaccurate", not so much so in more open country, but expensive would be a limitation everywhere. Many colonists, especilly on the frontier were people of extremely modest means.
You pick the strangest points on which to take a stand.