Hashing: How and Why to Check a File's Hash Value

Consider the following situation. You have been working for days on a PowerPoint presentation for work or school, and have been keeping the file on a shared computer, a network drive or even a personal flash drive. You put the final touches on your presentation the night before it’s due, save the file and get ready for a good night's sleep. The next day, you confidently begin your presentation. But imagine your surprise when you and your audience see the following image on your third slide:

You’ve been pranked. If you're lucky, everyone got a good laugh out of it. If not, there may be more serious consequences, depending on the situation. This sort of everyday  scenario raises an obvious question. Short of opening the file and manually perusing each slide in the presentation, how could you be sure that it had not been modified by any of the pranksters you may share your computer or network with? More seriously, how can we verify the integrity of a file that may or may not have been modified by a malicious individual seeking to infect out computer or network with a dangerous piece of malware?

In this article, we’ll consider these questions and discuss the pros and cons of one simple means by which we can verify a file’s integrity to ensure that it has not been tampered with, namely, by verifying its hash value. We’ll conclude with a quick tutorial on how to verify a file’s hash value on Mac, Linux and Windows systems, and provide some links to a few lectures on cryptographic hash functions culled from the series of courses listed in our collection of free online computer science courses. Our primary sources along the way will be Everyday Cryptography by Keith M. Martin, and Applied Cryptography by Bruce Schneier.

Malware comes in many different guises. As the Electronic Frontier Foundation writes in their Surveillance Self-Defense Project, malware is frequently spread by "trick[ing] the computer user into running a software program that does something the user wouldn't have wanted." Let's say you decide to download a file from a website you know and trust, and from which you have safely downloaded files in the past. How do you know, for example, that the file you have downloaded onto your computer is in fact the one intended by the trusted website? How do you know it was not altered in transit? How do you know it was not swapped for another file by a malicious attacker? And how can you determine this without running the file first? 

One simple way to verify a file's integrity is by confirming its hash value. In Everyday Cryptography, Martin writes: “Hash functions can be used to provide checks against accidental changes to data and, in certain cases, deliberate manipulation of data . . . As such they are sometimes referred to as modification detection codes or manipulation detection codes” (emphasis in original, Martin, p. 188). In our opening example, a suitable hash function would have allowed you to detect that your presentation had been modified in some way without ever opening it.

So, what is a hash function? The primary practical property of a hash function is that it compresses arbitrarily long inputs into a fixed length output (Martin, p. 189, Schneier, section 2.4). Furthermore, slight differences in the input data result in large differences in the output data. “A single bit change in the pre-image [i.e. the file you’re hashing] changes, on the average, half of the bits in the hash value,” (Schneier, section 2.4). Two of the most commonly used cryptographic hash functions are known as MD5 and SHA1. Schnier quotes NIST’s description of the SHA hash function as found in the Federal Register:

The SHA is called secure because it is designed to be computationally infeasible to recover a message corresponding to a given message digest, or to find two different messages which produce the same message digest. Any change to a message in transit will, with a very high probability, result in a different message digest. (Schneier, section 18.7.)

Here’s a simple example. I have created a plain text file named hello.txt on my Desktop. The file contains a single line that reads: “Hello there.” Applying the well-known sha1 hash function to the file produces the following hash value:
4177876fcf6806ef65c4c1a1abf464087bfbf337.

If I edit the file and remove the period from the end of the line so that it reads “Hello there”, the hash function now returns an entirely different value: 33ab5639bfd8e7b95eb1d8d0b87781d4ffea4d5d.

If I then return the file to its original state by adding the period back in to the end of the sentence, the hash value of the newly edited file will be the same as the original hash. And we would have seen much the same result (though it would have taken a good bit longer to compute!) if my original file had been a copy of the complete works of Shakespeare from which I then removed a period.  

Let’s consider a more practical example. The Electronic Frontier Foundation provides a number of recommendations on how to reduce your risk of malware infection in its Surveillance Self-Defense Project. At the top of their list, we read: “Currently, running a minority operating system [their examples are Linux and  MacOS -ed.] significantly diminishes the risk of infection because fewer malware applications have been targeted at these platforms. (The overwhelming majority of existing malware targets only a single particular operating system.)” This is more security through obscurity than anything else, but it’s still fun to try out new things, so after a bit of reading you decide to download a copy of the latest version of Ubuntu from an online repository.

How can you check to make sure that the file you’ve downloaded is the official one intended by Ubuntu’s developers and has not been manipulated or corrupted in transit? One way is to confirm that the file’s hash value is equivalent to the one provided by the developers. So you go to the page that lists the download’s hash value and make a note of it. Next, you run the hash function on the file you downloaded. If the resulting value is equivalent to the expected one, you have successfully verified the file’s hash.

However, it is critical to note here that verifying a file’s hash value by itself can only establish a relatively weak form of data integrity, in comparison with more robust mechanisms such as digital signature schemes which can provide a stronger form of integrity verification and even authentication. (Martin, pp. 186-189.) This is because a hash value such as we are discussing here cannot tell us anything about the origin of a digital file. For example, assume that unbeknownst to you, the site you’ve downloaded your file from has itself been compromised, and the attacker has: 1) replaced the download file with a piece of malware, and 2) also replaced the corresponding hash value that you use to check the file’s integrity with the hash value of the malware.

If you then verify the hash value of your downloaded file, you have done nothing more than verify the integrity of the malware! And you’re none the wiser because the site itself was compromised! At the same time, however, if you found out through another source that the site and file were compromised, you could then identify the malicious file and distinguish it from the legitimate source file. In a digital signature scheme, as mentioned above, the developer could digitally sign the legitimate hash value with a trusted key. In this way, the question of trust is then displaced to the question of signature authentication.

A second concern regarding this method of determining data integrity is the security of the hash functions themselves. There are known practical and theoretical vulnerabilities in two hash functions that are among the most common in use for these exact purposes on the web today: MD5 and SHA1. A discussion of these vulnerabilities is beyond the scope of the present article, but more information can be easily found online.

Still, as Bruce Schnier states, “we cannot use [one-way hash functions] to determine with certainty that the two strings are equal, but we can use them to get a reasonable assurance of accuracy.” (Schneier, section 2.4). In other words, hash functions can help us establish a basic level of data integrity. In our opening example, simply making a note of the hash and then checking it the next day would have sufficed to establish that the file had been tampered with. But, of course, if the file had been secured or encrypted to begin with, it never would have even been an issue in the first place.

Finally, how does one actually compute the hash value of a file? It is actually rather simple, but the specifics depend on your choice of operating system. MacOS and Linux systems come bundled with basic functionality to check any file’s hash value, while Microsoft Windows systems require you to download a piece of software to accomplish the task. Two of the most common functions used to verify file hashes are known as MD5 and SHA1. We’ll consider each in turn.

MacOS
1) Open up a command line Terminal.
2) Type “openssl md5 </path/to/file>” into the terminal and press enter.
2A) As an alternative to #2, you can also type “openssl md5 ” into the terminal, then drag and drop the target file into the Terminal window, and press enter.
3) The terminal will then return the MD5 hash value of the given file.

To compute the hash value of the file using a different hash function, type the name of that function into the terminal command in place of “md5”. For example, to compute the sha1 hash of a file, you would type: “openssl sha1 ” followed by the file path. To see a list of all the message digest commands available on your machine, type “openssl —help” into the command line terminal.

Linux (Debian-based)
1) Open up a command line Terminal.
2) Type: “md5sum </path/to/file>”. Then press enter.
3) The terminal will return the MD5 hash value of the given file.

To compute the hash value of the file using a different hash function, type the appropriate command into the terminal in front of the path to the target file. For example, “sha1sum </path/to/file>” will compute the file’s sha1 hash value. To see what other hash functions are available on your system, type “man dgst” into the terminal. 

Windows
Windows systems apparently do not come bundled with a built-in utility to check hash values. However, there are a number of different pieces of software you can download to accomplish the task. Microsoft Support lists the File Checksum Integrity Verifier, but warns that this is not supported by Microsoft and is only of use on Windows 2000, Windows XP and Windows Server 2003. This discussion at superuser provides a number of different extant options.

Video Lectures on Hash Functions

• Hashing with Chaining (MIT) 
• Table Doubling, Karp-Rabin (MIT)
• Open Addressing, Cryptographic Hashing (MIT)
• Hash Functions (Ruhr University, Bochum)
• SHA1 Hash Function (Ruhr University, Bochum)

As always, comments, questions, suggestions and angry tirades are welcome below.

Going Dark: New Email Technology To Easily Encrypt Everything

The unabashed abuse of privacy on the global scale is one of the most troubling invasions of our time. Now, one programmer is advocating a new paradigm of electronic communications that is simple and effective: "Dark Mail" that encrypts every email, every time.

Ladar Levison created the popular and secure email service Lavabit, which made news when he shut down the service entirely rather than cede to the goverment's demands that he surrender his security keys (which would have effectively undermined the entirety of the operation's purpose.) This happened hot on the heels of the Edward Snowden leaks, and since then no seriously secure single service has stepped forward to fill the gap. Now, as popularmechanics.com reports, Levison still wants to keep you covered.

In an interview posted today, Levison stated that everyone should be under the assumption that their electronic communications are being monitored at all times. This creepy but cruelly accurate statement is one that has yet to sink in for modern society, even though it means that everything from their (possibly "dangerous" and defamatory) private opinions to naked pictures are subject to scrutiny. He argues that the complexity of the e-communication infrastructure, coupled with the ease of cracking "endpoint security" (one's personal computer or device) makes things difficult for the average privacy-prone person. He has created "Dark Mail", a new encryption idea, to aid in spreading the powers of privacy.

As Levison explained:

"Dark Mail is really an effort to turn the world’s email dark—to make email encryption ubiquitous, universal, and automatic. The simplest explanation of what we’re doing is that we’re rewriting the protocols of email—the standard rules computers use for delivering email messages—so that messages are encrypted before they leave your computer and can’t be decrypted until they’ve reached the recipient’s computer. And because this is built into the system, there’s no cognitive burden. Grandma could use this—you don’t need to understand encryption or why it’s important. If someone can use email today, they will be able to use Dark Mail tomorrow."

Levison went on to elucidate that Dark Mail is not an email service, rather, it is a technology than any provider could implement. Expounding on PGP (Pretty Good Privacy) software, Dark Mail implements asymmetric cryptography techniques that use a public key (given to anyone who would like to send an automatically-message to a specific recipient) and a private key (theoretically, only the viewer of the message) to keep communications secure. Layers of anti-metadata technology to shake electronic position trackers are also in the works.

Levison went on to reference Phil Zimmerman, PGP's creator, and his lengthy police investigation and legal battle stemming from the creation of an encryption so strong that it was at first considered a munition (although the charges were eventually dropped.) On paper (and e-documents), there are laws that are in place to allow us this level of privacy.

Even if you feel you're doing nothing wrong, how do you know what those who would malign you are using against your favor? Why become a target just because you might be seeking knowledge that someone else deems illicit? Keep your privacy and your freedom close at hand, for both are valuable enough to be stolen.

If George W. Bush's personal oil painted nudies can be e-heisted, your info doesn't stand a chance. 

Cryptography vs. Cryptanalysis: Black Hat Talk on Hacking Tor Pulled from Conference

Historians of cryptology often describe the development of the discipline of "secret writing" in terms of a dialectic between cryptography and cryptanalysis, that is, between code makers and code breakers.  Cryptographers seek to create ever more indecipherable encryption schemes and cryptanalysts seek to break them.  An article on the cancellation of a Black Hat conference talk on the Tor privacy service from Reuters provides us with an interesting glimpse of how this tension is currently playing out among hackers and security researchers within the US government.  From Reuters, on the cancelled talk:

A highly anticipated talk on how to identify users of the Internet privacy service Tor was withdrawn from the upcoming Black Hat security conference, a spokeswoman for the event said on Monday.

The talk was canceled at the request of attorneys for Carnegie Mellon University in Pittsburgh, where the speakers work as researchers, the spokeswoman, Meredith Corley, told Reuters . . . a Carnegie Mellon attorney informed Black Hat that one of the speakers could not give the Tor talk because the materials he would discuss have not been approved for public release by the university or the Software Engineering Institute (SEI). . . .

Its abstract, titled "You don't have to be the NSA to Break Tor: De-Anonymizing Users on a Budget," had attracted attention within the security and privacy communities. The abstract had been published on Black Hat's website but has since been removed.

While the media often identify Tor as the preferred browser of child pornographers, criminal cartels and terrorist groups they often fail to note that it was actually originally developed by US government researchers. Or perhaps their implication is that the US government naturally falls under one of those umbrella terms? Whatever the case may be in that regard, the Reuters article hints at the state of the arms race between US government cryptographers and cryptanalysts. Another excerpt:

The U.S. government funded the creation and much of the operation of Tor as a communications tool for dissidents in repressive countries. But Tor has frustrated the U.S. National Security Agency for years, according to documents released by former agency contractor Edward Snowden.

That revelation has helped increase adoption by those seeking privacy for political reasons, as well as criminals, researchers say. 

Some criminal suspects on Tor have been unmasked by the U.S. Federal Bureau of Investigation and other law enforcement or intelligence agencies using a variety of techniques, including tampering with software often used alongside Tor.

Check out the EFF for more information on Tor.

Unintended Consequences of the Snowden Leak

In the aftermath of the Snowden NSA leak last year, supporters of the National Security Police State and Surveillance Society in the Republican and Democratic parties quickly ran to the media arguing that the leak represented a grave threat to national security because it would potentially reveal sensitive intelligence sources and methods.  It is now being reported by the WSJ that, in response to the Snowden leaks, Al Qaeda have changed up their crypto protocols and rolled their own encryption software.  And the uninformed  responses from the professional hysterics in the media and blogosphere are not hard to find.  Take some guy named Bob Cesca at the Daily Banter, for example. He writes:

So this is just peachy. I’ve always been very, very cautious to not over-emphasize the general scope of the terrorist threat, but this has more to do with stupidly and recklessly helping the ones that are out there, and it appears as if Snowden & Company have done precisely that.

What this reveals is that Bob Cesca doesn't know the first thing about the basic tenets of cryptography. One of the first things anyone learns when doing the most rudimentary study of cryptography and cryptanalysis is that "home-brewed," closed source  cryptographic software is essentially broken by definition, since by definition it cannot be subjected to rigorous review.  This makes it easier to break.  So, ironically, by switching up their crypto, Al Qaeda are likely providing new attack vectors for intelligence agencies the world over.  And this is in fact the view of at least one actual expert in cryptography, Bruce Schneier, as opposed to the uninformed reactions of professional political whiners.  He writes:

The Web intelligence company Recorded Future is reporting -- picked up by the Wall Street Journal -- that al Qaeda is using new encryption software in the wake of the Snowden stories. I've been fielding press queries, asking me how this will adversely affect US intelligence efforts.

I think the reverse is true. I think this will help US intelligence efforts. Cryptography is hard, and the odds that a home-brew encryption product is better than a well-studied open-source tool is slight. Last fall, Matt Blaze said to me that he thought that the Snowden documents will usher in a new dark age of cryptography, as people abandon good algorithms and software for snake oil of their own devising. My guess is that this an example of that.

Heartbleed: Critical OpenSSL Bug Exposes Secure Traffic

From Ars Technica:

Lest readers think "catastrophic" is too exaggerated a description for the critical defect affecting an estimated two-thirds of the Internet's Web servers, consider this: at the moment this article was being prepared, the so-called Heartbleed bug was exposing end-user passwords, the contents of confidential e-mails, and other sensitive data belonging to Yahoo Mail and almost certainly countless other services.

The two-year-old bug is the result of a mundane coding error in OpenSSL, the world's most popular code library for implementing HTTPS encryption in websites, e-mail servers, and applications. The result of a missing bounds check in the source code, Heartbleed allows attackers to recover large chunks of private computer memory that handle OpenSSL processes. The leak is the digital equivalent of a grab bag that hackers can blindly reach into over and over simply by sending a series of commands to vulnerable servers. The returned contents could include something as banal as a time stamp, or it could return far more valuable assets such as authentication credentials or even the private key at the heart of a website's entire cryptographic certificate.

Online Learning: Free Lecture Courses on Data Communications, Networking, Cryptography and Computer Security

I've been meaning to bring these resources together into a post for some time now.  There are a ridiculous number of free university level courses on communications, networking, cryptography and computer security available online.  Here are some of the better courses, lectures and video tutorials that I've come across over the last six months, all of which are appropriate for people who are looking for in depth introductions to these fields, or more experienced folks who would like a refresher on the fundamentals.

Lecture Series

• Steve Gordon's Lecture Courses
Steve Gordon is an Associate Professor at Sirindhorn International Institute of Technology (SIIT), Thammasat University, Thailand.  On his Youtube page, you can find four complete lecture series on Security and Cryptography, IT Security, Data Communications and Networks, and Internet Technologies and Applications. 

•  Introduction to Cryptography
Christof Paar, a Professor at Ruhr University, Bochum Germany, provides an introduction to modern cryptography in this series of 24 lectures. 

•  Cryptography and Network Security
Prof. D. Mukhopadhyay, from the Department of Computer Science and Engineering at the Indian Institute of Technology provides a broad introduction to Cryptography and Network security in this series of 41 lectures.  Production quality could be better, but the video lectures are substantive in nature.

•  Computer System Engineering
This undergraduate course, taught by Prof. Robert Morris and Prof. Samuel Madden from MIT, covers the basics of networking and computer security.  The first few lectures are not available.  But the units on networking and cryptography are available in full beginning with lecture 9.  

•  Fundamentals of Computer Networking 
This series contains over 30 lectures by Professor Parviz Kermani Department of Electrical & Computer Engineering at Manhattan College, and provides an in depth introduction to the basics of computer networking.

Miscellaneous Video

•  Whitfield Diffie on the History of Public Key Cryptography
•  Google Tech Talks on Cryptography (Assorted lectures and seminars from the Google Tech Talk series relating to cryptography and computer security)
•  Intro to Network Scanning (Basic introduction to network scanning tools)
•  Intro to Pentesting (10 short tutorials)

Schneier: "In the coming years we're seeing a lot more power struggles play out on the internet."

From an interview with Vice, Bruce Schneier speculates about how power struggles will play out on the internet in the coming years:

The internet is interesting because it really changes so many things. When the internet was born, there was this belief that it would vastly change the power structure. There's a great quote from John Perry Barlow in the mid-'90s at the World Economic Forum, and he basically says the governments of the world have no business on the internet, that have no power over the internet. You can't legislate it. The internet is it's own thing. It's a really utopian way of looking at the world, but we believed it. We believed the internet would change the world, would give power to the powerless. And it did, for many years. The ability to organize, to coordinate—it made so many things different.

And that changed recently. Governments discovered the internet. So now we're seeing that in China, for example, the internet is a tool of social control, and now both sides are using the internet. The Syrian dissidents are using the internet to organize, the Syrian government uses the internet to round up dissidents. That interplay between the institutionally powerful—the governments and corporations—and the distributively powerful—dissident groups, criminals, and hackers. How they both use the internet to increase their power, how they use the internet against each other, I think is fascinating. It's something that we need to look at. In the coming years we're seeing a lot more power struggles play out on the internet. And I'm just curious how that's gonna end up—it's not at all obvious.

Researchers Hack RSA with Acoustic Cryptanalysis

From CS.tau.ac:

Many computers emit a high-pitched noise during operation, due to vibration in some of their electronic components. These acoustic emanations are more than a nuisance: they can convey information about the software running on the computer and, in particular, leak sensitive information about security-related computations. In a preliminary presentation, we have shown that different RSA keys induce different sound patterns, but it was not clear how to extract individual key bits. The main problem was the very low bandwidth of the acoustic side channel (under 20 kHz using common microphones, and a few hundred kHz using ultrasound microphones), many orders of magnitude below the GHz-scale clock rates of the attacked computers.

Here, we describe a new acoustic cryptanalysis key extraction attack, applicable to GnuPG's current implementation of RSA. The attack can extract full 4096-bit RSA decryption keys from laptop computers (of various models), within an hour, using the sound generated by the computer during the decryption of some chosen ciphertexts . . .

Google Exec: "Encrypt Everything!"

From The Verge:

Since revelations of the NSA's widespread data collection and monitoring earlier this year, Google has staunchly denied working with the government agency and has taken it to task on a number of occasions. After calling the NSA surveillance "outrageous" earlier this month, Google executive chairman Eric Schmidt has come out against the agency again in an interview with Bloomberg News. "The solution to government surveillance is to encrypt everything," Schmidt said in a speed at the Johns Hopkins University School of Advanced International Studies. "We can end government censorship in a decade."

Police Pay Cryptolocker Ransom

From The Herald News:

A computer virus that encrypts files and then demands that victims pay a “ransom” to decrypt those items recently hit the Swansea Police Department.
The department paid $750 for two Bitcoins — an online currency — to decrypt several images and word documents in its computer system, Swansea Police Lt. Gregory Ryan said.

“It was an education for (those who) had to deal with it,” Ryan said, adding that the virus did not affect the software program that the police department uses for police reports and booking photos. . . .

CryptoLocker, a new Windows ransomware virus sweeping across the country, hit the Swansea Police Department on Nov. 6. The virus encrypted several files that could only be decrypted through the purchase of Bitcoins, an unregulated digital currency, to pay for the special “decryption key.” A countdown clock appeared on a computer screen showing how much time the department had to buy the key before all the files were deleted.

NSA Spying Revelations Boost Strong Crypto

Of course, there is no such thing as unbreakable encryption, but what do you expect from the corporate media?  From Fortune:

As revelations about the depth and breadth of the NSA's digital eavesdropping program continue to come to light, Ohio-based Battelle Memorial Institute is rolling out a new kind of network encryption designed to be virtually un-hackable -- not only now, but in the future. The non-profit research and development contractor has installed the first quantum key distribution (QKD) protected network in the U.S. linking its headquarters in Columbus to those in its manufacturing facilities in Dublin, Ohio, some 20 miles away.

Networking: 5 Wifi Securty Myths and the Crypto-Solution

PC World takes on some apparently popular wifi network security myths.  Excerpt:

Wi-Fi has evolved over the years, and so have the techniques for securing your wireless network. An Internet search could unearth information that’s outdated and no longer secure or relevant, or that’s simply a myth.

We’ll separate the signal from the noise and show you the most current and effective means of securing your Wi-Fi network . . . 

It concludes with a call for encryption:

Now that we’ve dispensed with five Wi-Fi security myths, let’s discuss the best way to secure your wireless network: encryption. Encrypting—essentially scrambling—the data traveling over your network is powerful way to prevent eavesdroppers from accessing data in a meaningful form. Though they might succeed in intercepting and capturing a copy of the data transmission, they won’t be able to read the information, capture your login passwords, or hijack your accounts unless they have the encryption key . . . 

Book Review: Hacking Secret Ciphers with Python

Hacking Secret Ciphers with Python is a free introductory textbook on cryptography, computer programming and the Python programming language written by Albert Sweigert, a software developer from San Francisco.  Hacking Secret Ciphers with Python is Sweigert's third book on Python, and the first that teaches the programming language through cryptography and traditional cryptographic protocols.

Published under a creative commons license, the work can be read for free online, downloaded as a .pdf or purchased from Amazon, with all proceeds going to the Electronic Frontier Foundation, Creative Commons and the Tor Project.  From the book's description:

“Hacking Secret Ciphers with Python” teaches complete beginners how to program in the Python programming language. The reader not only learns about several classical ciphers, but also how to write programs that encrypt and hack these ciphers. The full source code is given and explained line-by-line for ciphers such as the Caesar cipher, transposition cipher, simple substitution cipher, multiplicative & affine ciphers, Vigenere cipher, and hacking programs for each of these ciphers. The final chapters cover public key cryptography and the modern RSA cipher.

Clocking in at 416 pages, the book is broken down into 24 chapters covering virtually everything from the ancient Caesar Cipher to modern public key cryptography.  It thus provides a practical overview of the history of cryptography, while simultaneously introducing the reader to progressively more advanced aspects of the Python programming language. 

The book begins at the beginning, showing the reader first how to create rudimentary ciphers with paper and scissors.  It then gives a quick introduction on how to install Python, how to work with the interactive shell, and provides a quick overview of Python basics before jumping in to its first major coding chapter on the Reverse Cipher.  For each cipher covered in the book, it provides the Python code to run that cipher, followed by a chapter covering a second program that can be used to hack that cipher.  Python basics are covered in the analysis of the code used to create and then hack the given cipher.

Highly recommended for beginner to intermediate Python programmers who are interested in cryptography.  And since it is available free online, you can dive right in.

Research Group Cracks Taiwan's National "Smart Card" Digital Certificates

Ironically, it is often the ineptitude and incompetence of our security protocols that leave us the most insecure.  Is there anyone more vulnerable to attack that the person who thinks they are invulnerable because they have received reassuring platitudes and slogans from those running the security racket?  From Smart Facts, a report by a group of international researchers:  

An attacker can efficiently factor at least 184 distinct 1024-bit RSA keys from Taiwan's national "Citizen Digital Certificate" database. The big story here is that these keys were generated by government-issued smart cards that were certified secure. The certificates had all the usual buzzwords: FIPS certification from NIST (U.S. government) and CSE (Canadian government), and Common Criteria certification from BSI (German government).

These 184 keys include 103 keys that share primes and that are efficiently factored by a batch-GCD computation. This is the same type of computation that was used last year by two independent teams (USENIX Security 2012: Heninger, Durumeric, Wustrow, Halderman; Crypto 2012: Lenstra, Hughes, Augier, Bos, Kleinjung, Wachter) to factor tens of thousands of cryptographic keys on the Internet.

The remaining 81 keys do not share primes. Factoring these 81 keys requires taking deeper advantage of randomness-generation failures: first using the shared primes as a springboard to characterize the failures, and then using Coppersmith-type partial-key-recovery attacks. This is the first successful public application of Coppersmith-type attacks to keys found in the wild.

Google Seeks to Get in Ahead of NSA Scheme to Undermine Internet Encryption

From The Washington Post:

Google is racing to encrypt the torrents of information that flow among its data centers around the world in a bid to thwart snooping by the NSA and the intelligence agencies of foreign governments, company officials said Friday.

The move by Google is among the most concrete signs yet that recent revelations about the National Security Agency’s sweeping surveillance efforts have provoked significant backlash within an American technology industry that U.S. government officials long courted as a potential partner in spying programs.

Google’s encryption initiative, initially approved last year, was accelerated in June as the tech giant struggled to guard its reputation as a reliable steward of user information amid controversy about the NSA’s PRISM program . . . 

Netizen Self-Defense Against the NSA Adversary

Bruce Schneier literally wrote the book on Applied Cryptography.  In an article for the Guardian, provides some advice for those who are concerned about privacy and security and explains what measures he takes in order to secure his information.  From the Guardian:

I have five pieces of advice:

1) Hide in the network. Implement hidden services. Use Tor to anonymize yourself. Yes, the NSA targets Tor users, but it's work for them. The less obvious you are, the safer you are.

2) Encrypt your communications. Use TLS. Use IPsec. Again, while it's true that the NSA targets encrypted connections – and it may have explicit exploits against these protocols – you're much better protected than if you communicate in the clear.

3) Assume that while your computer can be compromised, it would take work and risk on the part of the NSA – so it probably isn't. If you have something really important, use an air gap. Since I started working with the Snowden documents, I bought a new computer that has never been connected to the internet. If I want to transfer a file, I encrypt the file on the secure computer and walk it over to my internet computer, using a USB stick. To decrypt something, I reverse the process. This might not be bulletproof, but it's pretty good.

4) Be suspicious of commercial encryption software, especially from large vendors. My guess is that most encryption products from large US companies have NSA-friendly back doors, and many foreign ones probably do as well. It's prudent to assume that foreign products also have foreign-installed backdoors. Closed-source software is easier for the NSA to backdoor than open-source software. Systems relying on master secrets are vulnerable to the NSA, through either legal or more clandestine means.

5) Try to use public-domain encryption that has to be compatible with other implementations. For example, it's harder for the NSA to backdoor TLS than BitLocker, because any vendor's TLS has to be compatible with every other vendor's TLS, while BitLocker only has to be compatible with itself, giving the NSA a lot more freedom to make changes. And because BitLocker is proprietary, it's far less likely those changes will be discovered. Prefer symmetric cryptography over public-key cryptography. Prefer conventional discrete-log-based systems over elliptic-curve systems; the latter have constants that the NSA influences when they can.

Since I started working with Snowden's documents, I have been using GPG, Silent Circle, Tails, OTR, TrueCrypt, BleachBit, and a few other things I'm not going to write about.

Is Any Private Data Safe from the Prying Eyes of Government?

Apparently not.  From the Guardian:

US and British intelligence agencies have successfully cracked much of the online encryption relied upon by hundreds of millions of people to protect the privacy of their personal data, online transactions and emails, according to top-secret documents revealed by former contractor Edward Snowden.

The files show that the National Security Agency and its UK counterpart GCHQ have broadly compromised the guarantees that internet companies have given consumers to reassure them that their communications, online banking and medical records would be indecipherable to criminals or governments.

The agencies, the documents reveal, have adopted a battery of methods in their systematic and ongoing assault on what they see as one of the biggest threats to their ability to access huge swathes of internet traffic – "the use of ubiquitous encryption across the internet".

The Data Security Defense Race

Are we in for a defense and arms race between tech firms or groups dedicated to user privacy and government or corporate entities that seek to undermine that privacy?  From Wired:

Technology companies are enabling security features that make certain types of government surveillance extremely difficult, and it’s a trend that’s set to continue. That’s why the U.S. government has long wanted laws that force tech companies to make their products wiretap friendly. . . .

In fact, advancements in the usability of cryptographic protocols have made anti-surveillance features relatively simple for technology companies to bake into their communications products. And public demand for greater security and privacy in the wake of Edward Snowden’s revelations may make it virtually obligatory for them to do so before new wiretapping laws can be introduced.

This heralds a looming standoff between technology companies and government . . . 

How Does a Password Hack Work?

A fairly well-detailed article at Ars Technica on the "Anatomy of a Hack" shows how hackers go about the process of cracking supposedly secure passwords.

In March, readers followed along as Nate Anderson, Ars deputy editor and a self-admitted newbie to password cracking, downloaded a list of more than 16,000 cryptographically hashed passcodes. Within a few hours, he deciphered almost half of them. The moral of the story: if a reporter with zero training in the ancient art of password cracking can achieve such results, imagine what more seasoned attackers can do.

Imagine no more. We asked three cracking experts to attack the same list Anderson targeted and recount the results . . . Even the least successful cracker of our trio—who used the least amount of hardware, devoted only one hour, used a tiny word list, and conducted an interview throughout the process—was able to decipher 62 percent of the passwords. Our top cracker snagged 90 percent of them.  

The strength and speed of this attack is not surprising however, since the passwords were encrypted with the MD5 algorithm, which is widely considered to be cryptographically broken.  The first flaws were found in the algorithm in the 1990's, and many more followed over the course of the last ten years.  So the question is: are a lot of websites still using broken encryption schemes?  And if so, how many? And which ones?

Secure Quantum Internet

Government researchers have revealed that they have been working on a cryptographically secure quantum internet for over two years.  From MIT Technology Review:

One of the dreams for security experts is the creation of a quantum internet that allows perfectly secure communication based on the powerful laws of quantum mechanics.

The basic idea here is that the act of measuring a quantum object, such as a photon, always changes it. So any attempt to eavesdrop on a quantum message cannot fail to leave telltale signs of snooping that the receiver can detect. That allows anybody to send a “one-time pad” over a quantum network which can then be used for secure communication using conventional classical communication.   . . . .

Today, Richard Hughes and pals at Los Alamos National Labs in New Mexico reveal an alternative quantum internet, which they say they’ve been running for two and half years. Their approach is to create a quantum network based around a hub and spoke-type network. All messages get routed from any point in the network to another via this central hub. . . .