Most non-browser HTTP clients do SSL / TLS wrong. Part of why clients do TLS wrong is because crypto libraries have unintuitive APIs. In this post, I’m going to write about my experience extending an HTTP client to configure Java’s Secure Socket library correctly, and what to look for when implementing your own client.
I volunteered to implement a configurable TLS solution for Play’s web services client (aka WS). WS is a Scala based wrapper on top of AsyncHttpClient that provides asynchronous mechanisms like Future and Iteratee on top of AsyncHttpClient and allows a developer to make GET and POST calls to a web service in just a couple of lines of code.
However, WS did not contain any way to configure TLS. It was technically possible to configure TLS through the use of system properties (i.e. “javax.net.ssl.keyStore”) but that brought up more messiness — what if you wanted more than one keystore? What if you needed clients with different ciphers? Sadly, WS isn’t alone in this: most frameworks don’t provide configuration for the finer points of TLS.
There is also a long and well known gulf between the security community and the developer community about the level of knowledge about TLS and the current state of the HTTPS certificate ecosystem. I want to fix that as well, and this blog post should be a good start.
So. Here’s what I did.
Table of Contents
For the sake of readability (i.e. avoiding TL;DR), I’m breaking this across several blog posts.
The pull request is on Github and you are invited to review the code and comment as you see fit.
In this blog post, I’m just going to cover the setup.
First, the problems that make TLS necessary.
- The First Problem: Programmers do not get security
- The Second Problem: Wifi / Ethernet is not secure
- The Third Problem: Man In the Middle
- Digression: Mitigation and General Security
- More Videos and Talks
Then, the implementation in WS.
- The Use Cases for WS
- Understanding TLS
- Understanding JSSE
- Configuring a client
- Debugging a client
- Choosing a protocol
- Choosing a cipher suite
Future posts will discuss certificates in more detail, but this gives us somewhere to start.
The First Problem: Programmers do not get security
The first problem is the assumption that TLS is overkill, built by researchers to protect against an abstract threat.
Unfortunately, this is not the case. TLS has real attacks against it, and they exist in the wild. Even worse, there are very serious, real world implications to breaking a TLS connection. Some people trust TLS in situations which could mean imprisonment or death.
But. Programmers work with bugs. Programmers get bugs. Programmers do not get security.
Programmers understand how bad input can ruin a programmer’s day. Programmers understand how corrupt data can completely ruin any hope of a functioning program. Programmers know that working with concurrency is so dangerous that it should only be done with special concurrency primitives and rules. Human users may be incompetent, but they are mostly benevolent: the forces working against the programmer are entropy and loose requirements.
Programmers don’t usually write programs that have to defend against an attacker. Most programmers have never even seen an attacker. Even the concept of a human deliberately trying to break or subvert a program is foreign. QA usually tests for successful cases, and maybe for some negative test cases… but QA typically doesn’t submit specially crafted XML documents that poke at the filesystem or chew up gigabytes of memory with character entities.
If programming is like driving a car, then the difference between working with QA versus working against an attacker is the difference between driving in rush hour versus driving with someone determined to run you off the road. TLS, and people using TLS based clients, have to assume that someone is going to try to run them off the road.
It also helps to see what an attack is like. For most programmers, an attack is theoretical, even a bad joke in poor taste. It doesn’t really become real until an actual attack is demonstrated by a security researcher in front of the programmer in question.
With that in mind, I’ve included several videos from real live security professionals in this blog post. You don’t have to watch them all at once, but you should watch them all eventually and see how they think.
The Second Problem: Wifi / Ethernet is not secure
TLS has a specific problem that most programmers do not have to deal with. TLS has to assume an attacker has access to the TCP/IP stream between client and server. This is commonly called packet snooping.
It is trivial to snoop on other computers in your network using tools like Wireshark. This is doubly true when using wireless networks, such as a coffee shop. People are often surprised that every single website they visit is being broadcast in the clear, in the same way as radio, but it’s true, and it’s easy to pick up those radio transmissions.
But don’t take my word for it. Here’s the Wifi Pineapple:
You can buy a Wifi Pineapple for $100 plus shipping.
It picks up all traffic sent over a wifi network. It’s so good at intercepting traffic that people have turned it on and started intercepting traffic accidentally.
Don’t think that WPA protects you from this. WPA2 is vulnerable to bruteforcing and most people choose passwords with extremely low entropy, partly because wifi passwords are shared so often.
The Third Problem: Man In the Middle
Not only can an attacker sniff packets on the network, but the attacker can also substitute traffic. Here’s a video of Cain & Abel at work:
Note that it takes less than 20 seconds to impersonate the server, after which the attacker can modify any URL coming from the server to point somewhere else. This is why rendering a login page in HTTP is essentially no protection at all: by the time the page is rendered, the attacker can make the HTML form send to a completely different URL.
This is not a theoretical attack. It has been automated to the point where rewriting web pages on the fly is fairly trivial — if you go down to your local hackerspace and browse the Internet without using a VPN, at some point you will find all your images URLs are pointing to Goatse.
Google has been subject to a host of attacks, from Iran. Note that the attack happened at the backbone, not at a particular coffee shop. Advanced persistent threats (APT) can include large nation states as well as script kiddies.
Encrypting data ensures that an attacker cannot read plaintext over the network, using public key encryption. However, the client still doesn’t know the identity of the server it’s trying to connect to. This is a problem. If you don’t verify the identity of the machine you’re talking to, then you could be talking to anyone.
Digression: Mitigation and General Security
Since first writing this, I’ve had some people for whom this has been their first exposure to just how insecure the Internet really is. So, before I head into some serious technical nerdery, I’d like to point out some good general resources for end users.
My Shadow shows the digital profiles that are exposed when you use the Internet, and shows how an attacker can leverage that information in aggregate. Often, rather than breaking encryption, the attacker will collect online facts about you, phone up tech support, and attempt to convince technical support to change your password.
Tactical Tech has a list of programs which are known to have reasonably good security.
Security in a Box discusses the operational security practices (also known as OPSEC) needed to use these tools effectively.
ONO Robot is a series of videos detailing how to use websites safely and securely (choosing good passwords, limiting cookies, etc).
Finally, Eva Galperin of the EFF gave a talk about guides to security in general, and what can be at stake for people who rely on these guides:
If this isn’t enough for you, then clicking through the following videos ought to change your mind.
Cryptography is a Systems Problem by Matthew Green;
The Use Cases
So that’s the threat model. Now the use cases for WS.
WS is a web services client, intended for asynchronous, non-blocking programmatic access to services using HTTP. Most clients will be RESTful with either a small (4k) XML or JSON payload or continously streaming data. Clients will only connect to a few well-known servers. Use of WS for general browsing or indexing a website is possible, but not the focus.
Client connects to internal WS service
In this use case, the client is talking to a service which is not publically available. The client and server will use private certificates (the “moxie option”), use a PKI management solution like DigiCert or OpenCA, or use an internal root CA.
The client may use mTLS / client authentication to connect to the internal service as an additional security measure.
The server will most likely support TLSv1.0. TLSv1.2 support is unlikely given that it does not come out of the box with ngnix and other clients. It is likely that the server supports RC4 ciphers.
Client connects to external WS service
In this use case, the client is talking to an external WS service, which is not owned by the organization and exists on the public Internet. The server may have a self-signed certificate, but is more likely to have a public certificate signed by a certificate authority.
Public facing “webscale” services using HTTPS are likely to support TLSv1.2 and support good ECC ciphers.
Client connects to public internet
In this use case, the client is calling up random URLs given to it by the web service and storing the content. This is a behavior of RSS feed web applications, which require connections to scrape and process data but do not typically analyse the contents.
The server could be anything. This is not the primary use case, and so the defaults will not be tuned for maximum compatbility with unknown or misconfigured servers.
This is going to be extremely abbreviated, but let’s give a refresher anyway. Adapted from Zytrax’s SSL Survival Guide (which also has some excellent sections on X.509 certificates):
TLS has four components: authentication, message integrity, key negotiation and encryption.
The client and the server begin a handshake.
The server sends a certificate and the chain of certificates leading back to a root certificate authority. The client should perform certificate and chain validation, making sure the chain terminates to a root CA trusted by the client.
In mutual TLS or client authentication, the client also sends a certificate to the server. This is rare, and most communication just authenticates the server to the client.
Because a server hands out the public key certificate and has the private key certificate, the client can encrypt all HTTP information using the public key and the server can decrypt it using the private key.
So far, so good. Next is JSSE.
JSSE is complex. The reference guide and the crypto spec are surprisingly helpful (once I started to understand it), but it wasn’t until I had the source code handy and could look at the internal Sun JSSE classes that I felt I had a handle on it:
- JSSE 1.7: Reference Guide, Crypto Spec, Sun Providers, Cert Path, Source code
- JSSE 1.6: Reference Guide, Crypto Spec, Sun Providers, Cert Path, Source code
In addition, the following best practices guides were very helpful:
It’s important to note that Play supports JDK 1.6. and 1.6 came out in December 2006. That’s over eight years ago. Since then, TLS (and the attacks on TLS) have evolved. Where possible, I wanted to bring 1.6 up to the 1.7 level of functionality, or at least note where it lags.
1 2 3
Wait, what? What’s a
TrustManager? What’s a
KeyManager? Well, from the JSSE Reference Guide (which is the first, last and frequently only word on the subject):
- TrustManager: Determines whether the remote authentication credentials (and thus the connection) should be trusted.
- KeyManager: Determines which authentication credentials to send to the remote host.
Most of the time, you’ll be working with a trust manager. You only need to worry about a key manager if you’re doing client authentication.
The interesting thing with this API, right off the bat, is that
init() takes null parameters for defaults, and it takes an array of managers.
What the JSSE Reference Guide says is “installed security providers will be searched for the highest priority implementation of the appropriate factory”. What actually happens is that you get an empty key manager and a default
X509TrustManagerImpl that points to
Likewise, the API takes an array of key managers, so you would expect this to work:
The problem here is that
init() doesn’t compose or aggregate managers together. As the Javadoc says, “only the first instance of a particular key and/or trust manager implementation type in the array is used. (For example, only the first
javax.net.ssl.X509KeyManager in the array will be used.)”
There is similar fineprint and tricksy assumptions throughout the JSSE API. If you don’t have the source code available, you will be utterly confused.
If you’re not using
SSLContext, then changes are done by setting system properties. This isn’t a bad way per se, but it’s global and opaque to the API.
Direct unit testing is painful as half the classes are defined as final, or use static methods. The internal logic is frustratingly and needlessly tightly coupled.
Despite all of this, and despite having interfaces temptingly near, you should NEVER replace the underlying JSSE functionality. Augment it, sure. Subclass away. Filter out weak points. But doing a straight up rewrite is a mistake. As per Moxie Marlinspike:
If you’re interested in writing a more restrictive TrustManager implementation for Android, my recommendation is to have your implementation call through to the system’s default TrustManager implementation as the very first thing it does. . That way you can ensure you at least won’t be doing any worse than the default, even if there are vulnerabilities in the additional checks you do.
Having read through TLS and JSSE, we’re now ready to check out how to configure the client.
Configuring a client
So, with the source code in hand, the first question was: how is WS?
It turns out that WS does the right thing. If you want to disable certificate validation, you have explicitly set the following in
which will let you accept a self-signed certificate that has not been added to your trust store.
However, the way
ws.acceptAnyCertificate is interesting. In Play 2.2.x, it looks like this:
1 2 3
That’s it. There’s no other logic that involves telling AsyncHttpClient to accept any certificate anywhere else in Play.
It turns out that accepting any certificate is the default behavior in AsyncHttpClient. If you are making HTTPS calls in Java using AsyncHTTPClient 1.7.x directly, you are vulnerable to a MITM attack.
The SSLContext class is central to the SSL implementation in Java in general and in AsyncHttpClient in particular. The default SSLContext for AsyncHttpClient is dependent on whether the javax.net.ssl.keyStore system property is set. If this property is set, AsyncHttpClient will create a TLS SSLContext with a KeyManager based on the specified key store (and configured based on the values of many other javax.net.ssl properties as described in the JSSE Reference Guide linked above). Otherwise, it will create a TLS SSLContext with no KeyManager and a TrustManager which accepts everything. In effect, if javax.net.ssl.keyStore is unspecified, any ol’ SSL certificate will do.
The first step in implementing HTTPS is to set up certificate verification to avoid issue 352. This, in itself, is fairly easy: just create an
SSLContext instance, then init with null values.
1 2 3 4 5
But, of course, that was only the beginning.
The essential problem with
ws.acceptAnyCertificate is while it’s wrong, it’s also a one line configuration setting. It’s obvious what it does. Meanwhile, the experience of adding a self signed certificate to the trust manager is downright painful. By default, the root CA certificates are in
$JAVA_HOME/lib/security/cacerts and so if you want to add an extra certificate (rather than replace all the existing CA certs), you have to know the exact keystore command for it:
Just from a deployment and maintenance perspective, this is a huge hassle. And it’s not like trust stores and keystores are all that complicated: there’s an list of certificates tied to aliases, with an optional password attached.
The simplest thing to do, from a programmer perspective, would be to have a list of stores that were pulled into a single manager. Then, instead of having to run a keytool command, you could just add a line saying where your store was, and you’d be done.
This involved creating a key manager and a trust manager that could take multiple stores. After looking through the source code, I determined that there was no API problem with using multiple stores inside a single manager… but then ran into the implementation again. In the X.509 implementation of JSSE, there’s a one to one correspondence between a manager and a store. I ended creating managers from the factories and then using a composite manager pattern based off Cody A. Ray’s blog post, and using the
X509TrustManagerImpl implementation and the
X509ExtendedTrustManager example as references.
The composite trust manager has a list of
X509TrustManagerImpl, and iterates through each one until it finds one that doesn’t throw an exception. If all of them throw exceptions, then it rethrows the exception with the entire list (so that no exceptions are swallowed), otherwise, it returns the first good result. This extends the
TrustManager functionality while safely keeping all of the existing logic in place.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34
Now you can now configure multiple key stores and trust stores directly in
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
and end up with a properly configured key manager and trust manager that contain all the keys from the various stores.
There’s a lot more to key stores and trust stores than I’ve mentioned here. For more details (including how to resolve improperly configured certificate chains), see:
- Java 2-way TLS/SSL (Client Certificates) and PKCS12 vs JKS KeyStores
- HTTPS with Client Certificates on Android
Configuring multiple clients
So now we have a configuration. But there’s another problem. There’s only one
application.conf file, and all the WS methods are on the companion object:
This meant that if you have several web services, say “secure.com” and “loose.com”, you cannot set up different configuration profiles for them, or set up a client dynamically, or do isolated testing. Everything had to be handled when the Play configuration loads.
I broke apart the
WS.client and added a
WSClient trait that could call
url in the same way. Now you can do this:
1 2 3 4 5 6 7 8 9 10 11 12
and have much finer grained control over the TLS configuration.
Unfortunately, getting a client passed as an implicit parameter to
WS.url is harder. I added a magnet pattern so you can do this:
1 2 3 4 5 6 7 8 9 10 11 12 13
and added another method
WS.clientUrl that takes an implicit client:
Debugging a client
While I was going through the client, I figured I may as well make it easier to turn on and off debugging as well.
Debugging is done by setting a system property, i.e.
-Djavax.net.debug="ssl". Debugging output is written directly to
System.out.println(), and the recommended way to change this is to change System.out. I can only hope this changes in JDK 1.8 — at the very least it should use
java.util.logging — but it’s what there is for now.
I had the JSSE debug page and the debug section of the reference guide handy, so it was fairly simple to provide that in configuration rather than futz with system properties. I added certpath and “ocsp” (an undocumented debug property) as well, while I was checking for certificate validation.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
This is not a perfect solution, because system properties are global across all clients. But it’s better.
ADDENDUM: this only worked intermittently and eventually I figured out why and fixed it. The more sensitive among you may wish to avoid this link.
Next, it was time to figure out what went into the client. The most important thing is the protocol.
Choosing a protocol
TLS comes in different versions. In JSSE, the list is available here.
There are two calls that refer directly to the protocol names, the getInstance call:
and the enabledProtocols list, which shows what the SSL context is willing to accept:
SSLv2 and SSLv2Hello (there is no v1) are obsolete and usage in the field is down to 25% on the public Internet. SSLv3 is known to have security issues and is still out in the field with 100% support. Virtually all HTTPS servers support it, and Mozilla Firefox still uses SSLv3 by default. They have a number of security issues compared to TLS.
TLSv1.2 is the current version, but early implementations of TLS 1.2 were prone to misconfiguration, which resulted in TLS 1.2 being disabled for the client by default in 1.7. Mozilla Firefox also has TLSv1.2 disabled, as of January 2014, and is only enabling it in the next version.
However, virtually all servers support TLS v1.0, and given our use cases, we expect that web services will have TLSv1.2 configured correctly. TLS 1.0 has been described as broken from the BEAST attack, but the attack seems to apply only to CBC ciphers, which we’re not obliged to use.
We want people to use the highest possible version of TLS. So we specify “TLSv1.2”, “TLSv1.1”, “TLSv1” in that order for JDK 1.7. For JDK 1.6, only “TLSv1” is available, so that’s what we have. We throw an exception on “SSLv3”, “SSLv2” and “SSLv2Hello”. If you don’t want that, then you have to explicitly set a
You can also specify the default protocol and the protocols list explicitly, i.e. if you want to configure JSSE for Suite B:
1 2 3 4 5 6
(Since this was written, JDK 1.8 came out and you can now set the system property “jdk.tls.client.protocols” to enable protocols.)
Next, it’s time to figure out what cipher suite to pick.
Choosing a cipher suite
A cipher suite is really four different ciphers in one, describing the key exchange, bulk encryption, message authentication and random number function. In this particular case, we’re focusing on the bulk encryption cipher.
Recommended Cipher Suites
In 1.8, the cipher list is ideal.
In 1.7, the default cipher list is reportedly pretty good.
In 1.6, the default list is out of order — some of the weaker ciphers show up before the stronger ciphers do. Not only that, but 1.6 has no support for Elliptic Curve cryptography (ECC) ciphers, which are much stronger and allow for perfect forward secrecy.
Now, the client doesn’t control what cipher will eventually be used. The server does. As a client, there are two things that you can do:
- You can present a list of ciphers which you are willing to accept.
- You can refuse a cipher which you know to be weak.
In 1.7, we use the default cipher list.
For 1.6, the client provides a truncated cipher list based off Brian Smith’s list, with the ECC ciphers taken out and the 3DES cipher removed. Roughly 55% of the Internet uses RC4, and given that WS is a web services client, it will probably be talking to only a few services which are current.
1 2 3 4 5 6 7 8 9 10
This isn’t the only possible option. There is an IETF recommended list of cipher suites:
and suggests TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 as preferred in general.
Deprecated Cipher Suites
There are some ciphers which everyone agrees are bad, and should never be used. NULL. Export Suite. DES. Anon. They are disabled by default, and the JSSE team says You are NOT supposed to use these cipher suites.
This brings up the next question: should WS consider RC4 and MD5 based ciphers to be weak? Surprisingly, probably not.
If you are setting up a server, you shouldn’t use RC4 and MD5 in your cipher suites, certainly. But if you’re a client, you’re probably fine talking to a server that is using RC4 or MD5.
In the case of RC4:
RC4 is horribly broken, and is horribly broken in ways that are meaningful to TLS. But the magnitude of RC4’s brokenness wasn’t appreciated until last year, and up until then, RC4 was a common recommendation for resolving both the SSL3/TLS1.0 BEAST attack and the TLS “Lucky 13” M-t-E attack. That’s because RC4 is the only widely-supported stream cipher in TLS. Moreover, RC4 was considered the most computationally efficient way to get TLS deployed, which 5-6 years ago might have been make-or-break for some TLS deployments. You should worry about RC4 in TLS —– but not that much: the attack is noisy and extremely time consuming. You should not be alarmed by MD5 in TLS, although getting rid of it is one of many good reasons to drive adoption of TLS 1.2.
The best, known attack against using RC4 with HTTPS involves causing a browser to transmit many HTTP requests — each with the same cookie — and exploiting known biases in RC4 to build an increasingly precise probability distribution for each byte in a cookie. However, the attack needs to see on the order of 10 billion copies of the cookie in order to make a good guess. This involves the browser sending ~7TB of data. In ideal situations, this requires nearly three months to complete.
The case of MD5:
The MD5 hash function is broken, that is true. However, TLS doesn’t use MD5 in its raw form; it uses variants of HMAC-MD5, which applies the hash function twice, with two different padding constants with high Hamming distances (put differently, it tries to synthesize two distinct hash functions, MD5-IPAD and MD5-OPAD, and apply them both). Nobody would recommend HMAC-MD5 for use in a new system, but it has not been broken.
The attacks on HMAC-MD5 do not seem to indicate a practical vulnerability when used as a message authentication code.
Note that we are talking about use of MD5 in a cipher here — as a client, accepting an MD5 signed certificate is a different kettle of fish.
Disabling Deprecated Cipher Suites
jdk.tls.disabledAlgorithms security property in 1.7 works fine to exclude bad or weak ciphers, and can also check small key sizes in a handshake. In particular, before 1.8, ephemeral DH parameters (DHE) were limited to 1024 bits, which is considered weak these days (although apparently still inherently stronger than RSA keys). You may also need to do this if you’re on 1.7, as there’s a nasty bug in 1.7 that causes connections to fail 0.05% of the time — although frankly, upgrading to 1.8 is a much better solution, just so you can specify “-Djdk.tls.ephemeralDHKeySize=2048” and get both perfect forward secrecy and a decent key size.
jdk.tls.disabledAlgorithms is a security property, not a system property, and is null by default:
The classes that use
TLSDisabledAlgConstraints, defined as static and final. There is no reliable and safe of setting the property dynamically in code — once the class has loaded, that’s what you’ve got.
Instead, you must set it in a properties file:
Once you’re done, reference that file from the command line using the undocumented java.security.properties system property:
Note that you will only be able to use ECC algorithms if you are on an Oracle JDK.
Another option is to set up the constraints on an SSLParameters object from setAlgorithmConstraints:
1 2 3 4
which looks much more convenient from a configuration perspective… but there’s a problem. The disabled algorithms filter is not supported in 1.6. Play supports 1.6, so if we want this feature, we have to do something else.
We can’t check the server handshake at runtime for the cipher, but we can cheat: we can check the SSLContext’s enabledCiphers list. We check the cipher list at configuration time, and throw an exception if we find that there is a weak cipher in the list. If you want to turn off the check, you have to configure the
I don’t think that ciphers need to be checked at run time, as I don’t think that the client will accept a cipher from the server that is not already in the client list.
As with protocols, you can configure the cipher list by hand:
1 2 3 4
If you have the option, you probably want to set
jdk.tls.disabledAlgorithms anyway: I don’t know of a way to check to ensure a minimum key size in the server handshake without using
And that about wraps things up for cipher suites.