The influence of Free Blacks was growing, not just those coastal Free Men of Color who owned slaves.
Actually, the influence of free blacks, particularly in Louisiana, was beginning to be restricted during the decade prior to the Civil War. Not only was manumission made more difficult (odd in a society hailed some on this board as individualistic to restrict how a person could deal with his or her property) but other restrictions against free blacks were passed. Some states, both in the North and South, banned free blacks from their state, seeing their influence as a threat.