One key realization the government came to during the PGP "hubub" going back to the 1990s was that most websites did not use encryption; in fact, most data everywhere was not encrypted. Going quiet was the most simple, obvious and impactful way for NSA to effect the result of the vast majority of data transmissions remaining "in the clear", thereby rendering decryption unnecessary for the bulk of data. It's interesting to note that early cell phones transmitted "in the clear" and suffered from susceptibility of user's account data being snooped out of the airwaves between cell phone and tower, until the cell phone companies adopted the practice of encrypting these transmissions. Of course, since the encryption keys are the phone company's and not the user's, the packets the NSA sees can be decrypted.
As encryption entered the public domain, however, the government had to recognize that it was no longer their own internal domain, but that many intelligent people around the world could and would be working on these endeavors outside of any government.
It was very much a case of the government's NSA needing to continue in the field full speed ahead to remain in the pack at the forefront. To not do this would mean being left behind and possibly be in a situation of having everyone being able to break NSA's encryption and NSA not being able to break anyone else's, a worst case - and inexcusable - scenario.
Since the math behind encryption is simply beyond most people, the public would always be using software written by a relatively small subset of the programming community. Proprietary encryption software is sold by a relatively small number of companies, and there's also all the open source software. A relatively small set of popular packages, as well as continually searching for new achievements that are being made and published, means that the only technology that NSA would be unfamiliar with would be technology developed without being published, which pretty much is limited to national intelligence services. And even those would be limited to those that chose to dedicate resources to the task, resources that would be content with carrying on their work without being published, but at the same time capable enough to achieve something that had not occurred to all published encryption researchers.
NSA pressed on and contributed security-related extensions to Linux in
SELinux. Mind you, this is open source, so the source code is published. Consequently, one could assume that the open source community (being very against the government controlling security technology unilaterally) would come out and publicly slam the NSA if their review of this code revealed any back doors.
SELinux is not about encryption per se, but I cite it as a topic to consider when considering NSA efforts, intentions and capabilities. Before assuming anything, one should gain much more perspective on the strategy and tactics employed by NSA historically. Perhaps one could start with the briefest of summaries of the
Wikipedia Cryptography entry, being sure to read the whole article, then do considerable research according to one's own interests in the subject.
As far as breaking the encryption that you've used in your transmissions, and thus being able to read your transmitted data, there would be two general categories of efforts to do that.
1) break the encryption in the sense of using encryption-breaking skill
2) defeat it through a "trick" of some kind
In the first case, NSA's history in the cryptography arena would strongly indicate, IMHO, that if a person unknowingly makes simple mistakes in their use of encryption that cracking it would be quite simple for NSA. There's little doubt that there would be many opportunities for individuals to make such mistakes. I could only assume that NSA would actively promote the spread of these opportunities using powerful if even simple techniques, such as promoting widespread use of old technology, for example, that was easily breakable.
In such cases, they would actually break the encryption, but they made some smart moves to help themselves in doing it.
In the second case, there's a virtual cornicopia of possible tricks that, if employed, would render decryption a trivial task. Probably the most powerful and most obvious (but I haven't thought about it much) would be to obtain users' private keys, but this category is really only limited by creativity. A good read of the NSA section of the above Wikipedia article and further research in those areas would reveal some possibilities. As a complex tool, encryption is one that requires the user to have a decent laymen's knowledge of it in order to succesfully use it.
It's also important to note, like others have on FR, that since such a small amount of data is encrypted, it stands out. Also, it can be saved for later use. Also, the NSA has plenty of processors with which to complete successful brute force attacks in selected cases, and they can limit their brute force efforts to only the very small percentage of data payloads that were transmitted between IP addresses that they are interested in. They certainly could automate this process, and have the ability to select certain IPs for constant decryption and both automated and manual analysis, as well as manually select certain traffic to process in this way as well.
Since analysis of associations between users and organizations can be used to establish lists of people to target, and does not require decryption, and can even use the presence of encryption as a flag of suspicion, it's certainly at least of equal concern as encryption, if not more, in terms of unconstitutional search and seizure.
If the website is using the https protocol, it will have a secure connection between the browsers of users browsing the website, and the webserver. So anyone sniffing (scanning) packets moving between the website and its user's browsers would see encrypted payloads.
If a website is publicly available on the internet, it can be crawled, interpreted and classified in an automated fashion.
This classification of a website need only be done once and reconsidered very infrequently, since the purpose and attitude of website publishers is basically static. In fact, web publishers purposely try to remain consistent in order to to attract users. So, though it sounds like a lot of processing, the website classification database, once built, is rather easily maintained, if the programmer working on it has reasonable skills.
Obviously the NSA crawls websites and classifies them, and notes the IP address they are hosted on, and puts this information into their internal database. While this crawling is executing, if a website is using https - that just means that the connection between the website and NSA's crawler is encrypted. Just like your browser automatically decrypts and displays the HTML pages, the NSA crawler sees the resulting HTML pages just fine.
If a website is public, at the end of the day using https matters very little. Of course, if the users are logged in and they have an expectation of privacy, such as when doing banking, https is essential.
Though a very simple technique, the practice of a) classifying IP addresses, domain names and URLs, b) analyzing user traffic by IP and c) then correlating and analyzing that data is a powerful one that does not appear to have Constitutional limitations on it so far.
Good points, some of which I've been saying in these discussions as they come up.
Though a very simple technique, the practice of a) classifying IP addresses, domain names and URLs, b) analyzing user traffic by IP and c) then correlating and analyzing that data is a powerful one that does not appear to have Constitutional limitations on it so far.
This is basic traffic analysis. Bruce Schneier writes about it a bit in Applied Cryptography.
One way to frustrate the classification methodogy that you wrote about is to have a public 'http' face (say, a website about unicorns or something), and a private 'https' face that concerns more sensitive topics. Granted, once FedGov or their minions managed to get a login to the site, they could crawl it, but if you had an organization that was very careful, and kept themselves small, (because any large organization will have government plants.), you could keep stuff reasonably private. I can actually think of a number of ways you could do some very interesting things with a set up like this, including dead-drops for messaging between cells.
The Constitutional aspect is more problematic, because the courts have stretched the concept of what are 'public areas' quite wide, mainly because we have no organ of government that is in the least bit interested in the liberty of the citizens of this country.For much of the NSA's history it was constrained by law to limit itself to foreign targets. Apparently, Big Brother wasn't satisfied with the FBI spying on every conceivable organization though, so they've brought the NSA into the "let's give Americans a big anal probe" game. Of course, all this spying through legal, extra-legal and sometimes plain outright unconstitutional methods is all done for our safety.
Sadly the brainwashed and miseducated citizens don't care enough to do much more than whine a little bit when some news does make it through in between american idol, the latest football game, or whatever else they are using as the circus to keep us all distracted from what the government is really up to.
Patrick Henry would have started shooting the bastards years ago. Sadly, to our great shame we've apparently decided to just live with it.
that or her knitting needles...