Closing the Open Door of Java Object Serialization
TL;DR
This is a long blog post, so please read carefully and all the way through before you come up with objections as to why it's not so serious. Here's the short version.
Java Serialization is insecure, and is deeply intertwingled into Java monitoring (JMX) and remoting (RMI). The assumption was that placing JMX/RMI servers behind a firewall was sufficient protection, but attackers use a technique known as pivoting or island hopping to compromise a host and send attacks through an established and trusted channel. SSL/TLS is not a protection against pivoting.
This means that if a compromised host can send a serialized object to your JVM, your JVM could also be compromised, or at least suffer a denial of service attack. And because serialization is so intertwingled with Java, you may be using serialization without realizing it, in an underlying library that you cannot modify.
To combat an attacker who has penetrated or bypassed initial layers of security, you need a technique called defense in depth.
Ideally, you should disable serialization completely using a JVM agent called notsoserial. This will give you a security bulkhead and you can add network monitoring to see if an attacker starts testing ports with serialized objects.
If you can't disable serialization, then there are options for limiting your exposure until you can remove those dependencies. Please talk to your developers and vendors about using a different serialization format.
The Exploit
If you can communicate with a JVM using Java object serialization using java.io.ObjectInputStream, then you can send a class (technically bytes that cause instantiation of a class already on the classpath) that can execute commands against the OS from inside of the readObject method, and thereby get shell access. Once you have shell access, you can modify the Java server however you feel like.
This is a class of exploit called "deserialization of untrusted data", aka CWE-502. It's a class of bug that has been encountered from Python, PHP, and from Rails.
Chris Frohoff and Gabriel Lawrence presented a talk called Marshalling Pickles that talked about some exploits that are possible once you have access to Java object serialization.
Practical Attacks
A blog post by FoxGlove Security took the Marshalling Pickles talk and pointed out that it's common for application servers to run ports with either RMI, or JMX, a management protocol that runs on top of RMI. An attacker with access to those ports could compromise the JVM.
The proposed fix was to identify all app servers containing commons-collections JAR and remove them.
The problem is, you don't need to have commons-collection running – there's a number of different pathways in. The ysoserial tool shows four different ways into the JVM using object serialization, and that's only with the known libraries.
There are any number of Java libraries which could have viable exploits. It isn't over.
Matthias Kaiser of Code White is doing further research in Exploiting Java Serialization, and says that more exploits are coming.
So, fixing this particular exploit doesn't fix the real problem, nor does it explain why it exists.
The Real Problem
The real problem is that "deserialization of untrusted input" happens automatically, at the ObjectInputStream level, when readObject is called.
You need to check untrusted input first before deserializing it, a process called "validation" or "recognition" if you're up on language security. But the specification is so powerful and complex that there isn't a good way to securely validate Java serialized objects.
Isolation – only having a management port open inside your "secure" data center – isn't enough.
Cryptography – using message authentication or encryption – isn't enough.
Obscurity – hoping that this bug is too obscure to be used by attackers or that your system is beneath their notice – isn't enough.
User level fixes – subclassing ObjectInputStream with a whitelist or wrapping your code in doPrivileged blocks – aren't enough.
I'll break all of this down in detail in the following sections.
EDIT: Charles Miller provides more context.
Why Isolation And Cryptography Aren't Enough
It's not a secret in the Java developer community that object serialization can load arbitrary classes. In fact, object serialization is the part of the reason that Java Applets are commonly disabled on browsers.
So, my first response to this exploit was "yes, this is why you don't publicly expose your management ports to attackers on the Internet." JMX and RMI are designed to work internally to the data center, and the assumption is that these ports are not publicly exposed. This is a commonly held tenet of server side applications – for example, Redis's security policy is explicitly “it’s totally insecure to let untrusted clients access the system, please protect it from the outside world yourself.”
To this end, developers typically assume that some combination of network isolation, sandboxing, containers and hypervisors will create a secure data center that will prevent attackers from gaining direct access to a port. In the event that ports need to be exposed outside the data center, there are options for RMI over TLS and JMX over TLS.
Rob Rodgers graciously and kindly corrected me on the problem inherent in this approach: it assumes that the attacker only ever attacks from the outside.
Firewalls are typically effective at preventing direct attacks from the outside, but they only cover one avenue of entrance. Rather than penetrate firewalls directly, attackers typically circumvent them by coming from another direction. They'll use the backdoor – connecting through unsecured laptops that manage payroll, desktop machines, smartphones, etc. If they're remote, they'll set up targeted phishing emails that have links that download malware. Or, if they're local, they'll try physical solutions – using wifi pineapple to get into the wireless network like TJ Maxx, dropping USB keys in the parking lot and so forth.
This sounds complicated, but to an attacker, it's about as simple as attaching a debugger to a running process and running through a breakpoint is to a developer. It's just part of the job.
Most attacks in the data center are from a compromised host inside the firewall. This changes the characteristics of the attacker: rather than a man in the middle, the attacker is now a man on the edge. Once there, they can see all the traffic to and from that machine. They could possibly poke at the DHCP server or the DNS server and start impersonating other clients. They have all the access to credentials that they need.
This is a class of exploit that you don't typically hear about in enterprise circles, because the whole idea is that there's an "inside" and an "outside" and the "inside" is secure. It's just not the case. The reality is that attackers typically manage to compromise one machine inside the corporate firewall, and then leverage that machine to gain access to others.
SSL/TLS doesn't help here. If you are in a coffee shop and connecting to a data center, TLS will protect you against a "Man in the middle" attack, where the user does not have the private keys necessary to break into the TLS session. If you are inside the data center, then things are different.
TLS will not save you. The calls are coming from inside the house. No combination of MAC, encryption and digital signatures can prevent a compromised host from sending a serialized nastygram.
For the same reason, obfuscation (what is commonly meant by "security by obscurity") is not a solution, because a compromised host will happy deobfuscate the data.
From the point of view of the attacker, the focus is on making the attack and attack vectors, which is why you don't typically hear the phrase "compromised host" all that often. Brian Keefer suggest “lateral movement.” Blake Hyde suggests "beachhead." Ben Tasker suggests "Pivot or Pivoting."
RMI has long been known to be a juicy target for hackers – it does not work well with firewalls or NAT, which is why many companies will run HTTP proxies through firewalls directly to RMI. There's a tutorial showing how to use Metasploit to gain access to RMI servers. And because JMX runs on RMI, there's every reason for operations to enable it and access it on a remote port.
You must assume that your network is insecure and that you may be talking to compromised hosts. You must establish bulkheads, a technique known as defense in depth to prevent the infection from spreading.
Why Obscurity Isn't Enough
You may be wondering why anyone would bother attacking you with this, especially if the attack is obscure and you don't think you're a big target. How real is this? Does anyone actually do this?
The problem is that while there aren't that many attackers over all, they benefit hugely from automation. Your average attacker is far more likely to be a script kiddie than he is to be Mr. Robot. The problem is that the script kiddies have libraries of every possible exploit, and automation frameworks that will run through every exploit until something works. There's even a Linux distribution known as Kali Linux, specifically written for systems penetration.
Security firms rely on "pen-testers" to audit companies for these vulnerabilities, and the industry has already added this one. Direct Defense put together a Burp Suite Extender called Super Serial with instructions on how to use it – this locates all serialized objects from the server in the Burp Suite network scanning tool.
The original exploit has already been refined: Trust Foundry has a blog post describing a "one click exploit" and has an executable jar file on Github.
EDIT: As of December 3rd, Impervia reports 645 attacks using this vulnerability from 503 different IPs.
So, relying on obscurity of the bug won't help: it's already been packaged and broadcast. And relying on your company's obscurity… well, attacks are common, cheap, and mostly automated. Maybe you'll be lucky for a while, but saying you won't be attacked is like saying you won't get spam in your inbox. It may be obscure to you, but it's not obscure to them.
And the end result? Well, look at Sony. Look at Target. Look at TJ Maxx. Look at Fandango and Credit Karma. Security breaches are real, and they have consequences.
The Fallacy of Trusted Input
The larger issue here is the implicit assumption that "untrusted input" implies that there can be trusted input. Really, it should be "unvalidated input." Trust – actual, no kidding, trust – is rare. You must trust the JVM class loader and your initial configuration files because you can't start the application without them, but everything after that is runtime input.
Barring actual custom hardware such as a trusted platform module or hardware security module, all runtime input can be compromised. Files can be rewritten. Network input can be spoofed. Anything that read in from I/O is "untrusted" in the sense that it has not been validated.
But even a object that has gone through validation is still untrustworthy: it could ask for things it has no right to, have faked credentials, etc. Trust or "untrust" are besides the point – input should have to jump some hurdles before it may be accepted for processing.
The Ideal Validation Scenario
If you are working in a microservices / domain driven design context, you build validation in an anti corruption layer around your bounded context (essentially a fancy word for "your app up to the point it has to work with I/O"). Everything outside the anti corruption layer is unvalidated. Everything inside the boundary is validated. This matches up well with language security principles. Here, we want full recognition before processing.
To this end, you use strongly typed objects called Value Objects to represent your validated input. As an example, the input to an "address service" might be "1 Market St, San Francisco CA 94111". This is a String – a raw, unvalidated, out of the box type. Raw types are broken. What you want is an Address – this means that you want the Address parser to validate and create a series of AddressLine, City and ZipCode objects. You never want to expose a raw type like String in your domain. You especially don't want to take raw types in your public APIs. If all your methods only use types created from validation, then you can limit your exposure. (Java suffers from not having value types, but they're showing up in JDK 1.9, finally.)
This is the ideal. In theory, you can take the raw bytes of every object that you received, parse and validate it, and only return an object after validation, instead of having ObjectInputStream create an object out of the blue.
Why Validation Is Hard
The reality is not so fun. ObjectInputStream will let you hook into it, and subclass it. The tough part is validating the input given that it's happening from inside ObjectInputStream.
Full on validation… turns out to be tricky. Very tricky. Sami Koivu has a couple of great blog posts in 2010(!) on why complex+powerful is a bad combination for security and breaking defensive serialization – I won't go into great detail, but the problem is that the serialization logic is so complex and powerful that "secure validation while deserializing is extremely difficult" and (according to Koivu) not only does the CERT guide not get it right, but even Josh Bloch doesn't get it right.
The Look-ahead Java deserialization solution that has been frequently mentioned suffers from this – the validation it suggests is a whitelist inside of ObjectInputStream.resolveClass – at this point you've already taken a bite out of the apple. Unless you are extraordinarily careful, even seemingly harmless whitelisted classes that are used in serialized objects can be used in an attack.
Why Whitelisting Isn't Enough
Whitelisting for allowed classes can prevent unknown classes from being called, but it can't prevent pathological classes from being deserialized. Wouter Coekaerts shows a denial of service attack using nested HashSets that would not be caught by whitelisting. Likewise, if you whitelist java.net.URL, you can't stop a specially constructed object doing a series of blocking network lookups based off the hashCode method.
Nor can whitelisting stop exploitation of underlying serialization bugs. It's a big, hairy piece of code, and there's no real way to slice off a piece of it.
You can, of course, implement your own custom parser and ignore ObjectInputStream completely. That will work for simple data objects, and you can throw away anything you don't recognize. That's a lot of work though, and of course the more complex the parser, the more likely it is to have bugs.
Why User Level Solutions Aren't Enough
You're probably thinking all is not lost.
You'll disable RMI and JMX ports, and move to JVM agent based solutions like Tapiki or Hyperic Sigar. Failing that, you'll install Jolokia or jmxtrans and try to limit your exposure.
You'll move to JSON (the lowest common denominator) or a faster, smaller, language independent binary protocol that does validation and schema resolution: something like Protocol Buffers / Capn Proto / Thrift / Avro. Or, if you're going to stick with Java, you'll use Kryo, although in that case I suggest adding Chill and having setRegistrationRequired turned on.
Then, you would carefully read the secure coding guidelines and follow all of the recommendations in your code, and you'd be fine.
The problem is that no matter what you do in your own code, you can't be sure that some library code, somewhere, isn't ignoring that completely and using a raw ObjectInputStream anyway. Github shows 678,489 instances of ObjectInputStream in various projects. JMX and RMI are only one vector – unless you decompile all your libraries and check them, you don't know that one of the libraries in your framework isn't using deserialization for a disk storage format (as was common in 200x), or for session backup, or for a distributed caching scheme.
Even then – even if you scan all of your code and can verify that ObjectInputStream.readObject is never called anywhere within your codebase – you still could be vulnerable. If you have any code that leans heavily on reflection (which happens more often than not in Java frameworks), then you can still have ObjectInputStream instantiated, although it will take more work to feed it the right bytes and call readObject. And I don't know what the impact is to sandboxed user supplied code that is supposed to run in a custom classloader, but I bet everyone will be reviewing code for a while.
EDIT: SRC:CLR has a list of 41 libraries that reference Apache Commons Collections and perform serialization, although they are careful to note they have not proved untrusted deserialization in all these libraries.
EDIT: Sijmen Ruwhof discusses scanning an enterprise organisation for the critical Java deserialization vulnerability. This is a useful way to find what systems have serialization exposed on their ports – either look for well known RMI / JMX ports and scan for the hexidecimal string "AC ED 00 05" or "rO0" – but it should be noted that this still only deals with known ports and vulnerable classes, and suggests depreciation of Java serializable objects long term.
Why SecurityManager Isn't Enough
There is something that is supposed to manage the JVM at a system level – SecurityManager. There are two problems with this: SecurityManager is both too strict and too permissive.
The system security manager has to be specifically enabled with -Djava.security.manager, and is a blunt instrument that causes lots of things to stop working everywhere. But SecurityManager doesn't limit ObjectInputStream in any significant way: you can limit implementations of subclasses of ObjectInputStream, but you can't make it safe or turn it off.
You can create a custom SecurityManager with your own permissions and set it on a thread, but you still have the same problem: any custom checks will apply to your user level subclasses of ObjectInputStream and any libraries can use the base ObjectInputStream with no restrictions.
Hacking the JVM
The ideal solution is to override ObjectInputStream itself, which will fix object serialization everywhere in the JVM.
My previous solution was to override the bootclasspath with a custom implementation of ObjectInputStream, but this contravenes the Java binary license. There is a precedence for using this method for fixing Sun's XML class bugs, but there is a better way.
The better way to do this, without violating the Java binary license, is to use Eirik Bjørsnøs's notsoserial project, a Java agent that will hook into the JVM and prevent and/or control deserialization everywhere.
Your Best Option
Your best option is to turn off object serialization completely, everywhere in the JVM, for good. This means no RMI, and no JMX, but there are other options.
Use notsoserial with nothing in the whitelist.
Your Second Best Option
In the event that you can't turn it off – find out what you can to squeeze the problem down until you can turn it off completely. If you do have reasons to have it on, then make sure that everything is logged, tracked and locked down as much as possible.
So, use notsoserial with whitelisting, by tracing your serialization. Keep network communication locked down with TLS client certificates, and investigate Jolokia or jmxtrans if you can.
Also, talk to your operations team about using Burp Suite or Haka to identify Java serialization across the network.
This might be a pain if you are using Java Mission Control / Java Flight Recorder in production.
Your Third Best Option
If you're writing or working with a library that requires object serialization and you don't have the option to control your clients or your endpoints, then you can work with a subclass of ObjectInputStream.
Take a look at ValidatingObjectInputStream or SerialKiller for an ObjectInputStream replacement.
Java's best option
The best option is for Java itself to remove serialization. Unfortunately, the last attempt JEP 154 was an April Fool's joke. There have been discussions to add an explicit serialization API, but I don't know of any current movement.
Another route that Oracle could take is to enhance the SecurityManager to allow more fine-grained control of serialization. This would work when the SecurityManager was enabled, but not break any existing classes or code otherwise.
Videos!
Wait, there's more!
Just because you're done with Java object serialization using ObjectInputStream doesn't mean that you're done: you can deserialize directly to classes using XMLEncoder as well!
Fortunately, now you get why deserialization is bad, you can use the same techniques shown here to validate or disable XMLEncoder in the same way. The point here is that you need to think about all the inputs to your system, and ask at each point if there is a gatekeeper. If you don't have validation built in, you're allowing an open door into your system.