Building Java Keystores

Building Java KeyStores is still a pain. I ran into this head on years ago, and even wrote some documentation but wanted to give it another try to see if I could simplify it.

In theory, this should be simple. There's only a few steps, after all:

1. Create certificates.
2. Make bundles!

In practice, it's a complex and fiddly process. Java has two managers involved in TLS. It has the KeyManager, which presents certificates. And it has the TrustManager, which receives presented certificates from the other end, and tells you if they are valid. You need to provide bundled information for each. They don't accept raw PEM files.

In JSSE, the name for a JKS or PKCS12 bundle is a keystore, which is the source of keys for a KeyStore instance. A "key store" is the keystore for the KeyStore used by the KeyManager, while a "trust store" is the keystore for the KeyStore used by the TrustManager, which does not contain keys. I hope that clears things up.

Whenever I want a keystore with no private keys, I say "trust store", so don't worry too much.

So, key stores need the private keys while trust stores need the public certificates.

1. Create certificates.
2. Put the public certificates in the trust stores.
3. Put the private keys with the public certificates in the key stores.

But server and client make different certificates, and so the key stores and trust stores have to be different between client and server. And you want a CA certificate backing everything. And you have to create certificate chains. So it's more like:

1. Create a CA certificate.
2. Create an EE server certificate signed by the CA.
3. Create an EE client certificate signed by the CA.
4. Create a server key store containing the server's private key and certificate chain.
5. Create a server trust store containing the CA's public certificate.
6. Create a client key store containing the client's private key and certificate chain.
7. Create a client trust store containing the CA's public certificate and any other CA certificates the client may need, e.g. $JAVA_HOME/lib/security/cacerts.

Palamino Labs has a nice overview of Java 2-way TLS/SSL (Client Certificates) and PKCS12 vs JKS KeyStores that goes into more detail.

But where it gets really complicated is that keytool doesn't let you import private keys directly. It will let you import keystores. It will let you generate keys with private keys. But it won't let you import a single private key and connect it with a public certificate.

There's a couple of approaches. The first one is to generate the key pairs in the keystore itself. That is, you use genkeypair, and then the private key is already there. This is the approach used for the Play TLS scripts.

# Create a server certificate, tied to example.com
keytool -genkeypair -v \
  -alias example.com \
  -dname "CN=example.com, OU=Example Org, O=Example Company, L=San Francisco, ST=California, C=US" \
  -keystore example.com.jks \
  -keypass:env PW \
  -storepass:env PW \
  -keyalg EC \
  -keysize 256 \
  -validity 385

However, this doesn't work great overall. In a real application you're more likely to have a real certificate authority that is offline, and you'll be dealing with intermediate CA certificates, and so on.

A more sustainable approach is to use Cloudflare's CFSSL to generate the certificates, and then work through various commands to create keystores. I used Drew Farris's sample-cfssl-ca script as a guide. You do need to install Go, but it's relatively painless to create the cfssl binaries.

export NAME=prototype
export ORG=myorg

cfssl gencert -initca ca-csr.json | cfssljson -bare $ORG-ca

cfssl gencert \
  -ca=$ORG-ca.pem \
  -ca-key=$ORG-ca-key.pem \
  -config=ca-config.json \
  -profile=server \
   server-csr.json | cfssljson -bare ${NAME}-server

cfssl gencert \
  -ca=$ORG-ca.pem \
  -ca-key=$ORG-ca-key.pem  \
  -config=ca-config.json \
  -profile=client \
   client-csr.json | cfssljson -bare ${NAME}-client

Once you've got the binaries created, you have to create the trust stores and key stores. Ideally you should be creating passwords for all of the relevant bundles. It's best if you create a custom password using pwgen or a secrets management tool.

export PW=`pwgen -Bs 10 1`
echo ${PW} > ${DIR}/password

And then in other scripts you can do:

export PW=`cat password`

Since I'm involved in importing and exporting anyway, I deliberately create PKCS12 and JKS for everything, which can be useful given older Java applications that have weak PKCS12 support.

#  Export server certificate chain with private keys to PKCS12
openssl pkcs12 -export \
  -passout env:PW \
  -inkey ${NAME}-server-key.pem \
  -name "$NAME-server" \
  -in ${NAME}-server.pem \
  -chain \
  -CAfile $ORG-ca.pem \
  -out ${NAME}-server-keystore.p12

# Import and create in JKS
keytool -importkeystore \
  -srckeystore ${NAME}-server-keystore.p12 \
  -srcstorepass:env PW \
  -alias "$NAME-server" \
  -srckeypass:env PW \
  -srcstoretype pkcs12 \
  -destkeystore ${NAME}-server-keystore.jks \
  -deststoretype jks \
  -deststorepass:env PW

Creating the trust store is done in the opposite direction, by importing the public certificate into the JKS keystore, and then converting to JKS using keytool.

Part of this is again historical, because in Java 7, PKCS12 wouldn't let you store certificate entries without a private key. But in practice, I had to do it this way, because I couldn't find a way to use openssl for this. When I tried importing the public key, it asked for -inkey. When I told it to use nokeys, it didn't import anything at all. The wiki page was not terribly helpful, and using keytool for both operations worked fine, so I stuck with it.

# Import the CA's public certificate in JKS
keytool -import \
  -alias $ORG-ca \
  -file  $ORG-ca.pem \
  -keystore ${NAME}-server-truststore.jks \
  -storepass:env PW << EOF
yes
EOF

# Convert from JKS to PKCS12
keytool -importkeystore \
  -srckeystore ${NAME}-server-truststore.jks \
  -srcstorepass:env PW \
  -srcstoretype JKS \
  -destkeystore ${NAME}-server-truststore.p12 \
  -deststoretype PKCS12 \
  -deststorepass:env PW

For the client, client authentication means that the client has to have a different private key than the server. Again, going from openssl PKCS12 keystore containing the private key, and importing the keystore using keytool:

# Export client certificate chain with private keys to PKCS12
openssl pkcs12 -export \
  -passout env:PW \
  -inkey ${NAME}-client-key.pem \
  -name "$NAME-client" \
  -in ${NAME}-client.pem \
  -chain \
  -CAfile $ORG-ca.pem \
  -out ${NAME}-client-keystore.p12

# Convert from PKCS12 to JKS
keytool -importkeystore \
  -srckeystore ${NAME}-client-keystore.p12 \
  -srcstorepass:env PW \
  -alias "$NAME-client" \
  -srckeypass:env PW \
  -srcstoretype pkcs12 \
  -destkeystore ${NAME}-client-keystore.jks \
  -deststoretype jks \
  -deststorepass:env PW

And then finally, for the client's trust store, the CA's public certificate and all of the cacert trust anchors must be added. You only need to do this if you're replacing the default trust store, but it's harmless.

# Import CA certificate into JKS
keytool -import \
  -alias $ORG-ca \
  -file  $ORG-ca.pem \
  -keystore ${NAME}-client-truststore.jks \
  -storepass:env PW << EOF
yes
EOF

# Import CA certs (assuming we have JDK instead of JRE here)
keytool -importkeystore
  -srckeystore $JAVA_HOME/jre/lib/security/cacerts \
  -srcstorepass changeit \
  -srcstoretype jks \
  -destkeystore ${NAME}-client-truststore.jks \
  -deststoretype jks \
  -storepass:env PW

# Convert from JKS to PKCS12
keytool -importkeystore \
  -srckeystore ${NAME}-client-truststore.jks \
  -srcstorepass:env PW \
  -srcstoretype jks \
  -destkeystore ${NAME}-client-truststore.p12 \
  -deststoretype pkcs12 \
  -deststorepass:env PW

I've written up the full script in doit.sh and it's a pull request against sample-cfssl-ca. If you run this, it requires no user interaction, and you can spend your time on something else.

Once you've got all the certificates, you'll want to set them on the JVM. The relevant system properties are in the JSSE guide tucked in the middle.

Best way to set it in an environment variable is with JAVA_TOOL_OPTIONS:

export JAVA_TOOL_OPTIONS="$JAVA_TOOL_OPTIONS -Djavax.net.ssl.keyStore=$NAME-server.keystore.jks"
export JAVA_TOOL_OPTIONS="$JAVA_TOOL_OPTIONS -Djavax.net.ssl.keyStorePassword=$PW"
export JAVA_TOOL_OPTIONS="$JAVA_TOOL_OPTIONS -Djavax.net.ssl.trustStore=server-truststore.jks"
export JAVA_TOOL_OPTIONS="$JAVA_TOOL_OPTIONS -Djavax.net.ssl.trustStorePassword=$PW"

The most common issue is that no client alias or the wrong client alias was picked out. One of the subtle things that can go wrong is to import only the server certificate into the key store, rather than the server's certificate chain, which includes intermediate and CA certificates as well. You need to add -chain -CAfile $ORG-ca.pem when you're exporting to PKCS12, and if you're using intermediate certificates then you can use -CApath with a directory of hashnamed links or files.

You can check that you're serving a certificate chain by pointing keytool at your server, i.e.

keytool -printcert -sslserver tersesystems.com

or if you want PEM format:

keytool -printcert -sslserver tersesystems.com -rfc

and then running it through Certificate Chain Composer.

If you want to see exactly where certificate validation went wrong, you need to set the java.security.debug system property, -Djava.security.debug=certpath, which will tickle the PKIX classes in the right way.

Use the Debug JSSE Provider which will show you if you've done it wrong and what certificates you have and check your results. If all else fails, use Wireshark.

You may note that the private key password is not shown as an option. That's because prior to 1.6, the key's password had to be the same as the key store's password, and the default on keytool is still to use the store password as the key password if it not specified. There's a number of ancient bugs attached to password protected keystores, but I won't bore you with the details. Just don't get fooled into thinking that passwords on keystores mean anything, or provide you with any security.

It is technically possible as of 1.8 to use a java.security.KeyStore.Builder, and the JSSE documentation is very keen to point out that "For example, it is possible to implement a Builder that allows individual KeyStore entries to be protected with different passwords. The javax.net.ssl.KeyStoreBuilderParameters class then can be used to initialize a KeyManagerFactory using one or more of these Builder objects." But you're still writing code to do it – it's not an option from the system properties.

If you want to aggregate keystores (as opposed to importing certificates a la cacerts above), then your options are limited. You can't specify multiple keystores as system properties, and both KeyManager and TrustManager will only work with a single keystore.

Java does have the "domain keystore" concept, but doesn't let you actually pick up a domain keystore from the commandline – you can't simply read in a keystore.dks file. Instead, as Pi Ke describes, you have to do something like this:

Map<String, KeyStore.ProtectionParameter> PASSWORDS =
        new HashMap<String, KeyStore.ProtectionParameter>();
PASSWORDS.put("keystore",
    new KeyStore.PasswordProtection("test123".toCharArray()));
PASSWORDS.put("policy_keystore",
    new KeyStore.PasswordProtection(
        "Alias.password".toCharArray()));
  PASSWORDS.put("pw_keystore",
    new KeyStore.PasswordProtection("test12".toCharArray()));
PASSWORDS.put("eckeystore1",
    new KeyStore.PasswordProtection("password".toCharArray()));
PASSWORDS.put("eckeystore2",
    new KeyStore.PasswordProtection("password".toCharArray()));
PASSWORDS.put("truststore",
    new KeyStore.PasswordProtection("changeit".toCharArray()));
PASSWORDS.put("empty",
    new KeyStore.PasswordProtection("passphrase".toCharArray()));

URI config = new URI(CONFIG + "#system");
KeyStore keystore = KeyStore.getInstance("DKS");
keystore.load(new DomainLoadStoreParameter(config, PASSWORDS));

But notice you have to provide a URL with DomainLoadStoreParameter, so you don't have the options of aggregating in-memory keystores – they have to be all on the filesystem or accessible through a URL or registered through a custom protocol handler, if you're desperate, and even then you can't chain domain keystores on top of one another, and the whole thing is defined through a custom format that's only defined in the Javadoc and has no BNF attached to it.

If you're using short lived certificates, i.e. something like Lemur will integrate with CFSSL, then you need to be able to update your keys. But if you want to change the private keys or certificates in an existing trust manager or key manager, you're in for problems.

First up: nothing is guaranteed to be thread safe. In particular, keystores are not thread safe. So you can't just point everything at a central keystore and mutate it. Also, the TrustManager and KeyManager implementations are essentially immutable after instantiation. They copy things internally, so even if you did point everything at a central keystore, they wouldn't use it.

The gory details: theoretically the Keystore.Builder lets you work with "dynamic keystores", but there are several problems there. The first is that it will only work with X509KeyManagerImpl, which corresponds to NewSunX509. So it's not the default algorithm for KeyManagerFactory, and it won't work with trust managers. The second is that it will delay creation of keystores, but won't let you change them after they've been created. The third is that KeyManagerParameters copies the list of Keystore.Builder internally so you won't be able to add or remove keystores to the KeyManager after initialization. The default trust manager sun.security.ssl.X509TrustManagerImpl is even less amenable, as it creates an internal collection out of the keystore at instantiation and does not allow external modification.

And if you want to get at the default trust store or key store to use as sources… you can't do it directly. The code for accessing the CA certs store and the system property defined key store is private, and very inaccessible. You can get to the CA certs store directly through the filepath, but you have to hit up the system properties for them yourself, which is awkward.

So the answer here is composite managers which give you finer grained control. You can use the CloudFoundry JSSE provider here, which contains a TrustManager and KeyManager that do file watching of keystores. And you can leverage the JVM default behavior by adding the system trust manager / key manager as the last elements in the composite. Whenever you have new keys that you need to add or remove, you load up a KeyManager or a TrustManager with a KeyStore and add it to the composite. One benefit to this approach is that if you have short lived certificates, you can put a KeyManager on a timer and have it removed automatically.

Putting certificates into keystores and loading new trust managers and key managers you will have to learn to do things like load X.509 certificates in Java:

KeyStore keystore = KeyStore.getInstance(KeyStore.getDefaultType);
keystore.load(null);
FileInputStream fis = new FileInputStream(filename);
CertificateFactory cf = CertificateFactory.getInstance("X.509");
Collection c = cf.generateCertificates(fis);
Iterator i = c.iterator();
while (i.hasNext()) {
  Certificate cert = (Certificate)i.next();
  keystore.setCertificateEntry(alias, cert);
}

and private key entries, which requires going through the certificate chain and setting a private key. The certificate chain is straightforward. The private key and password are not.

String alias = "my-server-key";
Key privateKey = ...; // SecretKeyFactory stuff
KeyStore keystore = KeyStore.getInstance(KeyStore.getDefaultType);
keystore.load(null);
Certificate[] internalCertificateChain = null;
CertificateFactory cf = CertificateFactory.getInstance("X.509");

try (FileInputStream inputStream = new FileInputStream(chainFile)) {
    CertPath certPath = cf.generateCertPath(inputStream);
    List<? extends Certificate> certList = certPath.getCertificates();
    internalCertificateChain = certList.toArray(new Certificate[]{});

    // Set a private key entry in the keystore
    char[] password = new char[0];
    keystore.setKeyEntry(alias, privateKey, password, internalCertificateChain);
} catch (CertificateException e){
    LOG.info("Tried and failed to parse file as a PKI :" + chainFile.getName(), e);
}

Getting a private key out of keystore requires a password for the private key entry – you can specify an empty char[] array, but it's odd that this is never mentioned in the documentation. Where it gets confusing is that the private key password is often assumed to be the same as the keystore password, i.e. in keytool, genkeypair assumes keypass is storepass if not specified.

Likewise, the default "SunX509" key manager assumes that all private keys use the same password:

char[] passwordForAllPrivateKeys = "".toCharArray(); // cannot be null
String algorithm = KeyManagerFactory.getDefaultAlgorithm(); // returns "SunX509" by default in 1.8
KeyManagerFactory kmf = KeyManagerFactory.getInstance("SunX509");
kmf.init(keyStore, passwordForAllPrivateKeys); // keystore is loaded by this point, so this password COULD be different than keystore password

If you're copying private keys between keystores, this assumption gets blown to heck. The best thing I can say is use "".toCharArray() everywhere for your internal keystores.

The situation does get a little better if you're using the "NewSunX509" key manager, which takes in parameters and uses the KeyStore.Builder API. This looks a bit like the DomainKeyStore code we saw above, but in this case it's applied to private keys, not keystores.

KeyManagerFactory kmf = KeyManagerFactory.getInstance("NewSunX509"); // note this must be defined explicitly

// First private key
KeyStore keystore1 = ... // this keystore contains only privatekey1
KeyStore.ProtectionParameter password1 = new KeyStore.PasswordProtection("password1".toCharArray())
KeyStore.Builder builder1 = KeyStore.Builder.getInstance(keystore1, password1);

// Second private key
KeyStore keystore2 = ... // this keystore contains only privatekey2
KeyStore.ProtectionParameter password2 = new KeyStore.PasswordProtection("password2".toCharArray())
KeyStore.Builder builder2 = KeyStore.Builder.getInstance(keystore1, password1);

List<KeyStore.Builder> builders = new ArrayList<>();
builders.add(builder1);
builders.add(builder2);
KeyStoreBuilderParameters builderParams = new KeyStoreBuilderParameters(builders);
kmf.init(builderParams); // now we can use different private key passwords!

Getting the secret key is simple if you have it in a keystore already. If not, things get more complicated. The "secret key factory" stuff has to be DER encoded. There's a PKCS8EncodedKeySpec for that, but if you have PEM encoding, i.e. you have a "BEGIN PRIVATE KEY" section around the DER encoding, then you have to strip it off and do it manually:

// optional keyPassword
Pattern KEY_PATTERN = Pattern.compile(
            "-+BEGIN\\s+.*PRIVATE\\s+KEY[^-]*-+(?:\\s|\\r|\\n)+" + // Header
                    "([a-z0-9+/=\\r\\n]+)" +                       // Base64 text
                    "-+END\\s+.*PRIVATE\\s+KEY[^-]*-+",            // Footer
            CASE_INSENSITIVE);

Matcher matcher = KEY_PATTERN.matcher(content);
if (!matcher.find()) {
    throw new KeyStoreException("found no private key: " + keyFile);
}
byte[] encodedKey = base64Decode(matcher.group(1));

if (!keyPassword.isPresent()) {
    return new PKCS8EncodedKeySpec(encodedKey);
}

EncryptedPrivateKeyInfo encryptedPrivateKeyInfo = new EncryptedPrivateKeyInfo(encodedKey);
SecretKeyFactory keyFactory = SecretKeyFactory.getInstance(encryptedPrivateKeyInfo.getAlgName());
SecretKey secretKey = keyFactory.generateSecret(new PBEKeySpec(keyPassword.get().toCharArray()));

Cipher cipher = Cipher.getInstance(encryptedPrivateKeyInfo.getAlgName());
cipher.init(DECRYPT_MODE, secretKey, encryptedPrivateKeyInfo.getAlgParameters());

return encryptedPrivateKeyInfo.getKeySpec(cipher);

Even things like getting a common name out of a X.509 certificate is a multi-step operation:

Stream.of(certificate)
    .map(cert -> cert.getSubjectX500Principal().getName())
    .flatMap(name -> {
        try {
            return new LdapName(name).getRdns().stream()
                    .filter(rdn -> rdn.getType().equalsIgnoreCase("cn"))
                    .map(rdn -> rdn.getValue().toString());
        } catch (InvalidNameException e) {
            log.warn("Failed to get certificate CN.", e);
            return Stream.empty();
        }
    })
    .collect(joining(", "))

This doesn't even get into the fun stuff like public key pinning and the various ways it goes wrong, or server name indication, or the limitations of the transport level identification that sslSession.getPeerCertificates() gives you. This is the basic stuff. and it's still complicated.

There is no reason that this is so hard. It is incidental complexity. It can be fixed, it can be papered over. And it's not you.