Depends on what state you're in. Here in North Dakota it is quite legal. Just not in Federal buildings, banks and the like.
Open carry is legal in Idaho. The Post Falls Police Department had to learn that the hard way after they repeatedly stopped an 18 year old carrying a Glock in a holster. Then his 13 year old brother got into the act carrying a .22 rifle (legally with written permission of his parents) on the street. While uncommon, it IS legal to carry openly here.