It's perfectly understandable that blacks and latinos would want to trace their geneology and figure out where they came from - so do many others. Census records have their limitations for all Americans, however. As you point out it's necessary to rely on birth, marriage and death records in many cases. Also useful are pension rolls for veterans of the Revolution and War of 1812; I don't think there are equivalents before the Revolution, although their might be British pension rolls for veterans of the Seven Years War (French and Indian War). In Europe, some families (such as mine) had private copies of records that are otherwise unavailable, and beyond that one must rely on church records, and the records of the various cities, duchies, principalities, and kingdoms. The various national (whether they have legal status or not) heraldic institutions can also be helpful.
There are some colonial era records--some organizations require documentation of an ancestor's service in the militia, public office, or whatever, to join. For Virginia, for example, there are militia records for some counties--but not for all of them. For specific counties records may have been lost when the court house burned (or in the case of Virginia, because of Richmond being set on fire in the closing days of the Civil War). I've read of one county in Virginia where Union troops took records from the court house and burned them.