The government can also infringe upon people's rights to life, liberty, property, freedom of speech, freedom of religion, freedom to bear arms, freedom to assemble, etc. when that person commits a crime and due process is followed. Does this mean that these can't be considered rights? Of course not!
You are a suspect in a crime -- did you commit one? What is the due process in investigating? The investigator can't ask questions of neighbors, co-workers etc.? Stake out a home or business? All that is invading privacy, and you're still considered an innocent person.