7.16.2 triggering Google Cloud security alert

Elastic Stack Elasticsearch
1

After bumping the docker image version to elasticsearch:7.16.2 in GKE, whenever I start elasticsearch pod, it immediately triggers a security alert Added Library Loaded

Added_Library_Fullpath: /tmp/twIW2T (deleted)
description: A library that was not part of the original container image was loaded. If an added library is loaded, this is a possible sign that an attacker has control of the workload and they are executing arbitrary code.
Process_Binary_Fullpath: /usr/share/elasticsearch/jdk/bin/java

This issue doesn't occur with 7.14.0 image version. Which I'm upgrading from.

2

I'm also seeing the following stack traces at the container startup. But not sure if it's related to the above added library.

{"type": "server", "timestamp": "2021-12-22T17:08:21,937Z", "level": "ERROR", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "es-cluster", "node.name": "es-cluster-data-2", "message": "failed to download database [GeoLite2-Country.mmdb]",
"stacktrace": ["org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];",
...
{"type": "server", "timestamp": "2021-12-22T17:08:21,939Z", "level": "ERROR", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "es-cluster", "node.name": "es-cluster-data-2", "message": "failed to download database [GeoLite2-City.mmdb]",
"stacktrace": ["org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];",
...
{"type": "server", "timestamp": "2021-12-22T17:08:21,936Z", "level": "ERROR", "component": "o.e.i.g.DatabaseNodeService", "cluster.name": "es-cluster", "node.name": "es-cluster-data-2", "message": "failed to download database [GeoLite2-ASN.mmdb]",
"stacktrace": ["org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];",
...
3

Yup it is probably related to that.

Perhaps read this..

Elasticsearch downloads the latest GeoIP databases.

You can set this.. but you should understand the impact

ingest.geoip.downloader.enabled : false

4

I would not expect downloading a GeoIP database to trigger this message, although I don't have any great alternative ideas either.

As a rule it's best not to investigate potential security issues in a public thread. Please follow the instructions on this page instead.

2 Likes
5

I am not as familiar with this exact error but I have worked with other containerization technology example Pivotal Cloud Foundry that would raise to his type of error because the expectation / policy is that container image is immutable and thus any changes / updates to the container would violate the security policy.

It may be something else instead.

@DavidTurner is correct please follow up using the procedure on the page he provided.

6

This is caused by the updated version of JNA and libffi.

Here's the code that writes that tmp file and memory maps it with executable flag: jna/tramp.c at 5.10.0 · java-native-access/jna · GitHub

This happens when a method is registered via JNA during call to ffi_closure_alloc.

(This is what is written there: jna/unix64.S at 5.10.0 · java-native-access/jna · GitHub)

3 Likes
7

The fact that a library was created and dynamically loaded isn't surprising, Elasticsearch has used JNA for many years which does this (perhaps not always, but it's always been a possibility). What's surprising is the filename: JNA's filenames typically contain the string jna and I thought libffi would include the string ffi too, but @henrist is right that it doesn't always.

1 Like
8

Today I Learned.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.