7.16.2 triggering Google Cloud security alert

After bumping the docker image version to Elasticsearch:7.16.2 in GKE, whenever I start Elasticsearch pod, it immediately triggers a security alert Added Library Loaded

Added_Library_Fullpath: /tmp/twIW2T (deleted)
description: A library that was not part of the original container image was loaded. If an added library is loaded, this is a possible sign that an attacker has control of the workload and they are executing arbitrary code.
Process_Binary_Fullpath: /usr/share/Elasticsearch/jdk/bin/java

This issue doesn't occur with 7.14.0 image version. Which I'm upgrading from.

I'm also seeing the following stack traces at the container startup. But not sure if it's related to the above added library.

{"type": "server", "timestamp": "2021-12-22T17:08:21,937Z", "level": "ERROR", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "es-cluster", "node.name": "es-cluster-data-2", "message": "failed to download database [GeoLite2-Country.mmdb]",
"stacktrace": ["org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];",
...
{"type": "server", "timestamp": "2021-12-22T17:08:21,939Z", "level": "ERROR", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "es-cluster", "node.name": "es-cluster-data-2", "message": "failed to download database [GeoLite2-City.mmdb]",
"stacktrace": ["org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];",
...
{"type": "server", "timestamp": "2021-12-22T17:08:21,936Z", "level": "ERROR", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "es-cluster", "node.name": "es-cluster-data-2", "message": "failed to download database [GeoLite2-ASN.mmdb]",
"stacktrace": ["org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];",
...

Yup it is probably related to that.

Perhaps read this..

Elasticsearch downloads the latest GeoIP databases.

You can set this.. but you should understand the impact

ingest.geoip.downloader.enabled : false

I would not expect downloading a GeoIP database to trigger this message, although I don't have any great alternative ideas either.

As a rule it's best not to investigate potential security issues in a public thread. Please follow the instructions on this page instead.

2 Likes

I am not as familiar with this exact error but I have worked with other containerization technology example Pivotal Cloud Foundry that would raise to his type of error because the expectation / policy is that container image is immutable and thus any changes / updates to the container would violate the security policy.

It may be something else instead.

@DavidTurner is correct please follow up using the procedure on the page he provided.

This is caused by the updated version of JNA and libffi.

Here's the code that writes that tmp file and memory maps it with executable flag: jna/tramp.c at 5.10.0 · java-native-access/jna · GitHub

This happens when a method is registered via JNA during call to ffi_closure_alloc.

(This is what is written there: jna/unix64.S at 5.10.0 · java-native-access/jna · GitHub)

3 Likes

The fact that a library was created and dynamically loaded isn't surprising, Elasticsearch has used JNA for many years which does this (perhaps not always, but it's always been a possibility). What's surprising is the filename: JNA's filenames typically contain the string jna and I thought libffi would include the string ffi too, but @henrist is right that it doesn't always.

1 Like

Today I Learned.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.