Log4j security vulnerability and plugins which bundle / vendor dependencies

[Kami (Tomaz)]

(this post has been moved from Zero-day-exploit in log4j2 which is part of elasticsearch - #25 by Kami)

Dear logstash community,

I would like to better understand on how log4j vulnerability affects logstash plugins which bundle / vendor their dependencies.

When auditing a logstash installation, I noticed multiple log4j jars bundled with various plugins:

find /opt/logstash/ -name "log4j*.jar"
/opt/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-http-3.4.2-java/vendor/jar-dependencies/org/apache/logging/log4j/log4j-api/2.11.1/log4j-api-2.11.1.jar
/opt/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.2.1-java/vendor/jar-dependencies/org/apache/logging/log4j/log4j-api/2.11.1/log4j-api-2.11.1.jar
/opt/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-azure_event_hubs-1.4.0/vendor/jar-dependencies/org/apache/logging/log4j/log4j-slf4j-impl/2.9.1/log4j-slf4j-impl-2.9.1.jar
/opt/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-azure_event_hubs-1.4.0/vendor/jar-dependencies/org/apache/logging/log4j/log4j-api/2.9.1/log4j-api-2.9.1.jar
/opt/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-test-0.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-core-5.6.4-java/lib/org/apache/logging/log4j/log4j-slf4j-impl/2.6.2/log4j-slf4j-impl-2.6.2.jar
/opt/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-test-0.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-core-5.6.4-java/lib/org/apache/logging/log4j/log4j-api/2.6.2/log4j-api-2.6.2.jar
/opt/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-test-0.2.0/vendor/bundle/jruby/2.5.0/gems/logstash-core-5.6.4-java/lib/org/apache/logging/log4j/log4j-core/2.6.2/log4j-core-2.6.2.jar
/opt/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.14.0.jar
/opt/logstash/logstash-core/lib/jars/log4j-api-2.14.0.jar
/opt/logstash/logstash-core/lib/jars/log4j-core-2.14.0.jar
/opt/logstash/logstash-core/lib/jars/log4j-1.2-api-2.14.0.jar
/opt/logstash/logstash-core/lib/jars/log4j-jcl-2.14.0.jar

After digging in further, I noticed that some logstash plugins bundle / vendor all of their dependencies in the RubyGem they publish (I assume that's done to make installations in airgap environments easier and / or similar).

After checking the gem metadata it shows that logstash plugin depends on logstash-core-plugin-api which depends on logstash-core which depends on log4j (so logstash-core is a transitive dependency of the plugin).

It appears that a lot of plugins transitively depend on older version of logstash-core which still requires log4j (logstash-core | RubyGems.org | your community gem host).

Newer versions of logstash-core don't seem to depend on it anymore (logstash-core | RubyGems.org | your community gem host - i assume they use log4j bundled in the logstash core, but dunno).

I would like to better understand if this is an indeed an issue and how to handle that in the problematic plugins - can we bump minimum version of logstash-plugin-core (and as such, logstash-core) or will this cause issues with logstash version compatibility? Or simply the best solution is not to bundle / vendor dependencies with the gem?

Thanks.

[lucas_weka (Lucas)]

We used GitHub - mergebase/log4j-detector: Detects log4j versions on your file-system, including deeply recursively nested copies (zips inside zips inside zips). to scan logstash and found:

/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.0.3-java/vendor/jar-dependencies/org/logstash/inputs/logstash-input-tcp/6.0.3/logstash-input-tcp-6.0.3.jar contains Log4J-2.x >= 2.0-beta9 (< 2.10.0) _VULNERABLE_ :-(

I'm not sure how this is used by logstash, but I removed the JndiLookup.class from that JAR for now. Maybe someone can clearify that.

[kakoni (Karri Niemelä)]

Typical logstash distributions (rpm, deb, docker ..) come with
logstash-input-azure_event_hubs, logstash-input-beats and logstash-input-http plugins, which all have dependency to log4j-api, but not log4j-core. As per current knowledge, log4j-api is not vulnerable.

[yaauie (Ry Biesemeyer)]

This is addressed in the ESA-2021-31 security announcement.

More details:

• Most plugins rely on log4j-api, which allows the plugins to use log4j provided by the application without providing their own. The log4j-api is not vulnerable to the CVE.
• 7.16.1 and 6.8.21 provide log4j-core 2.15, and also both still include an old, unreachable log4j-core in one of their bundled plugins (TCP input).
• The jar is unreachable (and therefore not exploitable) because of how plugins are loaded in Logstash, but removing its JNDI lookup class is a safe mechanism to add peace of mind
• updated versions of the TCP Input have been released to rely only on log4j-api and will be included in any subsequent patch releases.

[Kami (Tomaz)]

Thanks for the clarification.

This addresses all the concerns I had.

My main concern was with "The jar is unreachable (and therefore not exploitable) because of how plugins are loaded in Logstash, but removing its JNDI lookup class is a safe mechanism to add peace of mind".

I didn't dig through the plugin loading code so I wasn't sure if that bundled jar was actually being used or not, but now you clarified it's not being used.