Logstash 5.0.0-6.8.20 and 7.0.0-7.16.0: Log4j CVE-2021-44228, CVE-2021-45046 remediation

Announcements Security Announcements
1

Note — These instructions only apply if you are running Logstash 5.0.0 - 6.8.20, or 7.0.0 - 7.16.0. If you are running an older version of Logstash, or a version of Logstash >= 6.8.21 in the 6.x series or >= 7.16.1 in the 7.x series, these instructions do not apply. Please follow the guidance in main announcement

Instructions for removing JndiLookup from relevant JAR files​

These instructions only apply to users running Logstash versions between 5.0.0 and 6.8.20 (inclusive) or between 7.0.0 and 7.16.0 (inclusive). These must not be used in other versions of Logstash as there are safer, supported remediations (or no remediation is necessary).​

IMPORTANT: These instructions require manual changes to files within the Logstash installation. Such changes always carry a degree of risk. Elastic's recommendation is to upgrade to a supported version of Logstash in preference to manual patching.

Linux and MacOs Installations

We have provided a remediation script that will automatically remove JndiLookup.class from relevant jar files in a Logstash installation, backing up existing jar files before starting the process.

  1. These instructions must be executed as a user that has write access to the root directory for your Logstash installation (LOGSTASH_HOME) directory (and the files within it). The correct user will depend on how you installed Logstash. If you encounter Permission denied errors during this process, please check that you are logged in as the correct user.
    In some cases you may be required to use the sudo utility to perform a modification to the files. This is typically only necessary if the files are owned by the root user. If the files are owned by a regular user, the use of sudo may result in a change of ownership of those files, leaving them unreadable by the Logstash process.

  2. In a terminal window, change your working directory (cd) to the root directory for your Logstash installation. For installations from zip or tar files, this is the directory where you installed Logstash. For installations from RPM or Debian (.deb) packages, this will be /usr/share/logstash.

  3. Confirm that you are in the correct directory, and that the log4j file exists with:
    ls -l ./logstash-core/lib/jars/log4j-core-*.jar

  4. Copy script from https://ela.st/ls-rm-jndi-lookup naming it
    /tmp/remove_backup_jndi_lookup.rb

  5. On Logstash 6.0+ run
    bin/ruby /tmp/remove_jndi_lookup.rb

  • On Logstash 5.x.x run
    vendor/jruby/bin/ruby /tmp/remove_jndi_lookup.rb
  1. Confirm the removal of the vulnerable classes with:
    jar tvf logstash-core/lib/jars/log4j-core-*.jar | grep -i JndiLookup
    find . -name "logstash-input-tcp-*.jar"
    jar tvf <LOCATION OF JAR FILES FROM ABOVE STEP> | grep -i JndiLookup
  • If the removal has been successful, there will be no output.
  • If the output includes a listing for an entry in the JAR file, then the removal was not successful. Please review previous steps for any error messages.
  • Any other error is an indication of a problem and you should review your steps.
  1. Restart the Logstash node
  2. Repeat these steps for every Logstash node in your cluster

Docker Installations

To perform this in Docker, you can create a Dockerfile to include the command to remove the vulnerable JndiLookup class from relevant jars:

FROM docker.elastic.co/logstash/logstash:<LOGSTASH_VERSION>
RUN bin/ruby -rzip -e \
  'puts Dir.glob(["**/*/logstash-input-tcp-*.jar", "**/*/log4j-core*.jar"]).each \
  {|zip| puts zip; Zip::File.open(zip, create: true) \
  {|zipfile| zipfile.remove("org/apache/logging/log4j/core/lookup/JndiLookup.class") }\
  }'

To verify removal, shell into the newly created docker container, and run the following:
jdk/bin/jar tvf logstash-core/lib/jars/log4j-core-2.15.0.jar | grep Jndi
find . -name "logstash-input-tcp-*.jar"
jdk/bin/jar tvf <LOCATION_OF_ABOVE>| grep JndiLookup

*Manual Remediation On Linux or MacOS​

These instructions are provided in the unlikely situation that the provided script is unable to remediate the situation on Linux or MacOS.

These instructions assume the existence of the zip utility on your operating system.

  1. Confirm that zip is available by entering zip in a terminal window. If the output includes copyright text and a list of options, then the "zip" utility is available. If the output is an error message such as
    command not found: zip or
    The program 'zip' can be found in the following packages

  2. If no zip utility is present on your operating system, then you need to install it. Please consult your operating system vendor's documentation for installing packages.

  3. These instructions must be executed as a user that has write access to the root directory for your Logstash installation (LOGSTASH_HOME) directory (and the files within it). The correct user will depend on how you installed Logstash. If you encounter Permission denied errors during this process, please check that you are logged in as the correct user.
    In some cases you may be required to use the sudo utility to perform a modification to the files. This is typically only necessary if the files are owned by the root user. If the files are owned by a regular user, the use of sudo may result in a change of ownership of those files, leaving them unreadable by the Logstash process.

  4. In a terminal window, change your working directory (cd) to the root directory for your Logstash installation. For installations from zip or tar files, this is the directory where you installed Logstash. For installations from RPM or Debian (.deb) packages, this will be /usr/share/logstash.

  5. Confirm that you are in the correct directory, and that the log4j file exists with:
    ls -l ./logstash-core/lib/jars/log4j-core-*.jar

  6. If there is an error, then one of the following may be true:

  • you are in the incorrect directory - check that you are in the Logstash home directory
  • you have a typo in your command - check that you have entered it exactly as above
  • you are already on an up to date version of Logstash (6.8.21+ or 7.16.1+) that does not contain this vulnerable version of the log4j JAR file - check your Logstash version

  1. Make a backup of the vulnerable log4j JAR file with:
    zip ./backup-log4j.zip ./logstash-core/lib/jars/log4j-core-*.jar

  2. Follow this slightly amended process for the tcp input jar, which may also include a vulnerable version of the log4j code.

  3. Find the version of the tcp input jar that needs to be patched from the same directory as previously:
    find . -name "logstash-input-tcp-*.jar"

  4. This should give the location of the tcp input jar in your system, and should look something like:
    ./vendor/bundle/jruby/2.5.0/gems/logstash-input-tcp-6.2.1-java/vendor/jar-dependencies/org/logstash/inputs/logstash-input-tcp/6.2.1/logstash-input-tcp-6.2.1.jar

  5. Make a backup of this file
    zip ./backup-tcp-input.zip $(find . -name "logstash-input-tcp-*.jar" -print)

  6. These backup files are useful if you make a mistake in any of the subsequent steps, or experience any problems with the updated JAR files. You can safely delete them once you are confident that your Logstash node is working correctly.
    Note: It is important that you follow the above command as written. Other methods for backing up the jar file risk having it included on the Logstash classpath during execution, which may prevent your node from starting correctly.

  7. Remove the vulnerable class from your log4j JAR file with:
    zip -d ./**/*/log4j-core-2.*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

  8. Note: If running the above command produces a series of lines that look like :
    zip warning: Local Version Needed To Extract does not match CD: org/apache/logging/log4j/core/util/WatchManager$WatchRunnable.class

this has no effect on the removal of the vulnerable class and can safely be ignored.

  1. If the output is deleting: org/apache/logging/log4j/core/lookup/JndiLookup.class, then the command has been successful

  2. If the output is zip error: Nothing to do!, it means the class has already been deleted

  3. Any other error is an indication of a problem and you should review your steps.

  4. Follow the same process for your tcp input JAR file with:
    zip -d ./**/*/logstash-input-tcp-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

  5. Confirm the removal of the vulnerable classes with:
    jar tvf logstash-core/lib/jars/log4j-core-*.jar | grep -i JndiLookup
    jar tvf <LOCATION OF JAR FILE FROM PART 9> | grep -i JndiLookup

  • If the removal has been successful, there will be no output.
  • If the output includes a listing for an entry in the JAR file, then the removal was not successful. Please review previous steps for any error messages.
  • Any other error is an indication of a problem and you should review your steps.
  1. Restart the Logstash node
  2. Repeat these steps for every Logstash node in your cluster
3 Likes