Building Java KeyStores is still a pain. I ran into this head on years ago, and even wrote some documentation but wanted to give it another try to see if I could simplify it.
In theory, this should be simple. There's only a few steps, after all:
- Create certificates.
- Make bundles!
In practice, it's a complex and fiddly process. Java has two managers involved in TLS. It has the
KeyManager, which presents certificates. And it has the
TrustManager, which receives presented certificates from the other end, and tells you if they are valid. You need to provide bundled information for each. They don't accept raw PEM files.
In JSSE, the name for a JKS or PKCS12 bundle is a keystore, which is the source of keys for a
KeyStore instance. A "key store" is the keystore for the
KeyStore used by the
KeyManager, while a "trust store" is the keystore for the
KeyStore used by the
TrustManager, which does not contain keys. I hope that clears things up.
Whenever I want a keystore with no private keys, I say "trust store", so don't worry too much.
So, key stores need the private keys while trust stores need the public certificates.
- Create certificates.
- Put the public certificates in the trust stores.
- Put the private keys with the public certificates in the key stores.
But server and client make different certificates, and so the key stores and trust stores have to be different between client and server. And you want a CA certificate backing everything. And you have to create certificate chains. So it's more like:
- Create a CA certificate.
- Create an EE server certificate signed by the CA.
- Create an EE client certificate signed by the CA.
- Create a server key store containing the server's private key and certificate chain.
- Create a server trust store containing the CA's public certificate.
- Create a client key store containing the client's private key and certificate chain.
- Create a client trust store containing the CA's public certificate and any other CA certificates the client may need, e.g. $JAVA_HOME/lib/security/cacerts.
Palamino Labs has a nice overview of Java 2-way TLS/SSL (Client Certificates) and PKCS12 vs JKS KeyStores that goes into more detail.
But where it gets really complicated is that keytool doesn't let you import private keys directly. It will let you import keystores. It will let you generate keys with private keys. But it won't let you import a single private key and connect it with a public certificate.
There's a couple of approaches. The first one is to generate the key pairs in the keystore itself. That is, you use
genkeypair, and then the private key is already there. This is the approach used for the Play TLS scripts.
However, this doesn't work great overall. In a real application you're more likely to have a real certificate authority that is offline, and you'll be dealing with intermediate CA certificates, and so on.
A more sustainable approach is to use Cloudflare's CFSSL to generate the certificates, and then work through various commands to create keystores. I used Drew Farris's sample-cfssl-ca script as a guide. You do need to install Go, but it's relatively painless to create the cfssl binaries.
Once you've got the binaries created, you have to create the trust stores and key stores. Ideally you should be creating passwords for all of the relevant bundles. It's best if you create a custom password using
pwgen or a secrets management tool.
And then in other scripts you can do:
Since I'm involved in importing and exporting anyway, I deliberately create PKCS12 and JKS for everything, which can be useful given older Java applications that have weak PKCS12 support.
Creating the trust store is done in the opposite direction, by importing the public certificate into the JKS keystore, and then converting to JKS using keytool.
Part of this is again historical, because in Java 7, PKCS12 wouldn't let you store certificate entries without a private key. But in practice, I had to do it this way, because I couldn't find a way to use openssl for this. When I tried importing the public key, it asked for
-inkey. When I told it to use
nokeys, it didn't import anything at all. The wiki page was not terribly helpful, and using
keytool for both operations worked fine, so I stuck with it.
For the client, client authentication means that the client has to have a different private key than the server. Again, going from
openssl PKCS12 keystore containing the private key, and importing the keystore using
And then finally, for the client's trust store, the CA's public certificate and all of the cacert trust anchors must be added. You only need to do this if you're replacing the default trust store, but it's harmless.
I've written up the full script in doit.sh and it's a pull request against sample-cfssl-ca. If you run this, it requires no user interaction, and you can spend your time on something else.
Once you've got all the certificates, you'll want to set them on the JVM. The relevant system properties are in the JSSE guide tucked in the middle.
Best way to set it in an environment variable is with
The most common issue is that no client alias or the wrong client alias was picked out. One of the subtle things that can go wrong is to import only the server certificate into the key store, rather than the server's certificate chain, which includes intermediate and CA certificates as well. You need to add
-chain -CAfile $ORG-ca.pem when you're exporting to PKCS12, and if you're using intermediate certificates then you can use
-CApath with a directory of hashnamed links or files.
You can check that you're serving a certificate chain by pointing keytool at your server, i.e.
or if you want PEM format:
and then running it through Certificate Chain Composer.
If you want to see exactly where certificate validation went wrong, you need to set the
java.security.debug system property,
-Djava.security.debug=certpath, which will tickle the PKIX classes in the right way.
Use the Debug JSSE Provider which will show you if you've done it wrong and what certificates you have and check your results. If all else fails, use Wireshark.
You may note that the private key password is not shown as an option. That's because prior to 1.6, the key's password had to be the same as the key store's password, and the default on
keytool is still to use the store password as the key password if it not specified. There's a number of ancient bugs attached to password protected keystores, but I won't bore you with the details. Just don't get fooled into thinking that passwords on keystores mean anything, or provide you with any security.
It is technically possible as of 1.8 to use a
java.security.KeyStore.Builder, and the JSSE documentation is very keen to point out that "For example, it is possible to implement a Builder that allows individual KeyStore entries to be protected with different passwords. The
javax.net.ssl.KeyStoreBuilderParameters class then can be used to initialize a KeyManagerFactory using one or more of these Builder objects." But you're still writing code to do it – it's not an option from the system properties.
If you want to aggregate keystores (as opposed to importing certificates a la
cacerts above), then your options are limited. You can't specify multiple keystores as system properties, and both
TrustManager will only work with a single keystore.
Java does have the "domain keystore" concept, but doesn't let you actually pick up a domain keystore from the commandline – you can't simply read in a
keystore.dks file. Instead, as Pi Ke describes, you have to do something like this:
But notice you have to provide a URL with DomainLoadStoreParameter, so you don't have the options of aggregating in-memory keystores – they have to be all on the filesystem or accessible through a URL or registered through a custom protocol handler, if you're desperate, and even then you can't chain domain keystores on top of one another, and the whole thing is defined through a custom format that's only defined in the Javadoc and has no BNF attached to it.
If you're using short lived certificates, i.e. something like Lemur will integrate with CFSSL, then you need to be able to update your keys. But if you want to change the private keys or certificates in an existing trust manager or key manager, you're in for problems.
First up: nothing is guaranteed to be thread safe. In particular, keystores are not thread safe. So you can't just point everything at a central keystore and mutate it. Also, the
KeyManager implementations are essentially immutable after instantiation. They copy things internally, so even if you did point everything at a central keystore, they wouldn't use it.
The gory details: theoretically the
Keystore.Builder lets you work with "dynamic keystores", but there are several problems there. The first is that it will only work with
X509KeyManagerImpl, which corresponds to
NewSunX509. So it's not the default algorithm for
KeyManagerFactory, and it won't work with trust managers. The second is that it will delay creation of keystores, but won't let you change them after they've been created. The third is that
KeyManagerParameters copies the list of
Keystore.Builder internally so you won't be able to add or remove keystores to the
KeyManager after initialization. The default trust manager
sun.security.ssl.X509TrustManagerImpl is even less amenable, as it creates an internal collection out of the keystore at instantiation and does not allow external modification.
And if you want to get at the default trust store or key store to use as sources… you can't do it directly. The code for accessing the CA certs store and the system property defined key store is private, and very inaccessible. You can get to the CA certs store directly through the filepath, but you have to hit up the system properties for them yourself, which is awkward.
So the answer here is composite managers which give you finer grained control. You can use the CloudFoundry JSSE provider here, which contains a TrustManager and KeyManager that do file watching of keystores. And you can leverage the JVM default behavior by adding the system trust manager / key manager as the last elements in the composite. Whenever you have new keys that you need to add or remove, you load up a
KeyManager or a
TrustManager with a
KeyStore and add it to the composite. One benefit to this approach is that if you have short lived certificates, you can put a
KeyManager on a timer and have it removed automatically.
Putting certificates into keystores and loading new trust managers and key managers you will have to learn to do things like load X.509 certificates in Java:
and private key entries, which requires going through the certificate chain and setting a private key. The certificate chain is straightforward. The private key and password are not.
Getting a private key out of keystore requires a password for the private key entry – you can specify an empty
char array, but it's odd that this is never mentioned in the documentation. Where it gets confusing is that the private key password is often assumed to be the same as the keystore password, i.e. in keytool,
storepass if not specified.
Likewise, the default "SunX509" key manager assumes that all private keys use the same password:
If you're copying private keys between keystores, this assumption gets blown to heck. The best thing I can say is use
"".toCharArray() everywhere for your internal keystores.
The situation does get a little better if you're using the "NewSunX509" key manager, which takes in parameters and uses the
KeyStore.Builder API. This looks a bit like the
DomainKeyStore code we saw above, but in this case it's applied to private keys, not keystores.
Getting the secret key is simple if you have it in a keystore already. If not, things get more complicated. The "secret key factory" stuff has to be DER encoded. There's a
PKCS8EncodedKeySpec for that, but if you have PEM encoding, i.e. you have a "BEGIN PRIVATE KEY" section around the DER encoding, then you have to strip it off and do it manually:
Even things like getting a common name out of a X.509 certificate is a multi-step operation:
This doesn't even get into the fun stuff like public key pinning and the various ways it goes wrong, or server name indication, or the limitations of the transport level identification that
sslSession.getPeerCertificates() gives you. This is the basic stuff. and it's still complicated.
There is no reason that this is so hard. It is incidental complexity. It can be fixed, it can be papered over. And it's not you.