The government (state in this case) has the power to make and enforce laws. They decide who gets to drive (age limit, legal status, etc.), how often you have to qualify (take tests, eye exams), and what the penalties are for breaking the laws. Absolutely - they have the power and responsibility to take a driver's privileges away if the driver demonstrates his inability to respect the laws. People lose their licenses all the time. When you apply for and receive a license, you agree to abide by the laws. It's simple.
That said, we have the right to elect our officials and change the laws we don't agree with. The people wrote the State Constitution and the laws and the people can change it, so yes, "We can take it away at any time."
It's my opinion that government largely ignore and control the population, as opposed to working for us, so when I heard about government giving and taking priviledges, I sorta snapped. I hadn't finished my coffee yet.