Dealing with log4shell aka CVE-2021-44228 aka the log4j version 2

Quick note: from now on I will refer to log4j version 2 as “log4j2”

Update note: This blog entry was updated Dec 17, 2021, to include a note about the second remote code execution vulnerability CVE-2021-45046 in log4j

Okay if you haven’t heard about the critical vulnerability in log4j2 then I envy you. The Canadian Minister of National Defence issued a public statement about it, on a Sunday, and as a Canadian, trust me when I say this: my government working on a Sunday means it’s really bad.

Long story short: a remote code execution flaw exists in log4j2, which is used by basically every Java application on the planet (it’s the de facto logging library). It’s remotely exploitable, and not just through the front end, but on the back ends. Essentially, if you use log4j2 with default settings, any user-controlled string can be used to exploit it (so web browser user agent, username, iPhone name, the list is endless). If you want to keep up to date on #log4shell you’re going to have to track Twitter and GitHub, more on this in the second article in this series. I’m also intentionally ignoring how this vulnerability came to be, for now, that post mortem will be part three in this series (the bad news: this type of flaw is very very common and not what you expect). Update note: the fix of course had a flaw, which was initially labelled as a moderate Denial of Service (DoS) but was then found out to allow remote code execution.

So what do we do now? I’m going to give some advice you’re expecting, and some you aren’t.

First off, I’m going to say something unpopular: because of how many Java applications are built, especially using build systems like Maven, there’s a good chance the source code is either lost, or you never had it in the first place, and you may not be able to update all instances of log4j2. So my advice is going to take this realistic but painful truth into account.

Detection of the use of log4j2

One technique that works reasonably well is to assume anything written in Java in the last decade, that logs data, is using log4j2 and vulnerable. I’m sort of joking, but I’m sort of serious. A more useful response is to use tooling and scripts to detect instances of log4j2, if you use any SBOM/CI/CD type software that claims to detect what you’re actually building and allows you to block old versions, and known vulnerable versions, then it’s time to make sure your vendor has added support for this issue. There are also numerous ad-hoc techniques using grep on Linux/UNIX, Powershell on Windows, and so on. Please see the links at the end of this blog entry, taken from GSD-2021-1002352.

Updating log4j2

Special note: The Apache Software Foundation fix for CVE-2021-44228 was found to contain another security vulnerability, CVE-2021-45046. Initially, CVE-2021-45046 was labelled as a moderate Denial of Service) DoS but it was then discovered several days later that it did in fact allow for remote code execution. At this time any attempts to update log4j to 2.15.0 or later need to be the latest version, which is a moving target as always.

Your three main options with dealing with log4j2 are:

1. Keep or update to version 2.15.0 or later
2. Keep or update to a version between 2.10 and 2.14.1
3. Keep using a version between 2.0 and 2.9.1

The first option is of course the “ideal” solution, upgrade to 2.15 or later ensures the code is safe by default (addendum: 2.16 and later may remove it entirely, more on this in part three of this blog series). So if you can update to this version and deploy it, great, you’re done, and you can probably keep the updates rolling longer term which is also great (You didn’t think this is the last major vulnerability in log4j2 did you?).

The second option is less ideal but still functional. Log4j2 added a switch (set “log4j2.formatMsgNoLookups” to “true”) in version 2.10 that allows disabling of message pattern converter lookups. For reference, this was released in November of 2017. This requires a restart of the application and further verification to ensure it actually worked (more on this later).

The third option is of course the least desirable, but if you can update the source code, but not update the version of the library, you can add “%m {nolookups}” in all the pattern definitions to prevent lookups.

Okay, so those were the ideal scenarios where you can update the software. Let’s assume a common situation where you have a commercial product, and the vendor hasn’t updated it yet. Or maybe they went out of business. Or maybe you simply want to ensure future problems are less severe.

Workaround 1 - remove the jar files

Simply removing the jar files is an option, this will of course break logging via log4j2. Also, depending on how deeply the jars are embedded you may miss them. This is probably the worst workaround possible as it is intrusive and prone to error.

Workaround 2 - firewalling

I used to place outgoing firewall rules on all my web servers to prevent attackers from establishing reverse backdoor shells (I of course also had inbound firewalls to prevent traditional backdoor shells from being made available). This will work to some degree for vulnerable servers with log4j2, and more importantly, you should be doing this anyway. One note: if the server can make DNS lookups, then attackers can still scan for vulnerable instances of log4j2 which will trigger the DNS lookup, but having a firewall should block the outgoing connections of an actual attack.

Workaround 3 - attack and hotfix your servers

Normally the thought of using exploit code to attack, compromise and then modify running servers to address a vulnerability is simply too extreme and we won’t consider it.

Welcome to how bad the log4j2 vulnerability is.

There are two OpenSource projects taking similar but slightly different approaches to hotfix this:

Log4jHotPatch

"This is a tool which injects a Java agent into a running JVM process. The agent will attempt to patch the lookup() method of all loaded org.apache.logging.log4j.core.lookup.JndiLookup instances to unconditionally return the string \"Patched JndiLookup::lookup()\". It is designed to address the CVE-2021-44228 remote code execution vulnerability in Log4j without restarting the Java process. The dynamic and static agents are known to run on JDK 8 & 11 on Linux whereas on JDK 17 only the static agent is working (see below)"

Logout4Shell

"However, enabling these system properties requires access to the vulnerable servers as well as a restart. The Cybereason research team has developed the following code that exploits the same vulnerability and the payload therein forces the logger to reconfigure itself with the vulnerable setting disabled - this effectively blocks any further attempt to exploit Log4Shell on this server."

Conclusion

This is a really bad security vulnerability fixes and solutions that we normally wouldn’t consider are on the table. I’m not suggesting people should add dodgy-looking exploit and hotfix solutions to their standard operations but this is a special case, (and I acknowledge I am arguing from authority here) and as someone who has processed over 20,000 vulnerabilities over 20+ years and issued over 6000 CVE identifiers: this is by far the worst vulnerability I have seen in my career. Also, corporations like Amazon AWS are also advocating for customers to use tools like Log4jHotPatch. We’re living in a brave new world.

Also as an addendum, I’ve gone through several hundred tweets and several hundred links to find you the best ones:

Best list of vulnerable software: https://github.com/NCSC-NL/log4shell/tree/main/software

Best list of vulnerable services: https://github.com/YfryTchsGD/Log4jAttackSurface

Best hotpatch: https://github.com/corretto/hotpatch-for-apache-log4j2

Best detection:

• grep: https://gist.github.com/Neo23x0/e4c8b03ff8cdf1fa63b7d15db6e3860b
• jarhashes: https://github.com/mubix/CVE-2021-44228-Log4Shell-Hashes
• semgrep: https://github.com/returntocorp/semgrep-rules/pull/1650/commits/ecfc32623eec718d61ec83b9196574f333191008/
• yara: https://github.com/timb-machine/log4j/

Scanners:

• burpsuite: https://github.com/silentsignal/burp-log4shell
• Nmap NSE: https://github.com/Diverto/nse-log4shell
• Assorted: https://github.com/alexbakker/log4shell-tools and https://github.com/fullhunt/log4j-scanhttps://github.com/takito1812/log4j-detect

I think it’s safe to say if you’re relying on CVE, you’re missing out (none of the above links are in the CVE yet, unless you count a link to my tweet that links to the GSD file that I added the above to). Part two will go into how to stay up to date during a crisis.

Final update note:

The bad news: you get to redo all the work you did with CVE-2021-44228. The good news: you already know how to handle CVE-2021-45046, just repeat what you did for CVE-2021-44228.

Read the next blog in this series, Keeping up with log4shell aka CVE-2021-44228 aka the log4j version 2, to get a rundown of how #log4shell unfolded.