Repository S3 plugin amazon_client_exception

[mmarinm (Marin Mestrovic)]

Hi, I need some help with the issue I am getting when I try to take a snapshot with repository s3 plugin. I can't figure out what is the reason for the error. I don't know Java. But as far as I could understand it seems there is an issue with some internal java issues for the version of java that runs in the elastic container.

Please help me troubleshoot this.

I am using elasticsearch 5.6.4 and repository-s3 5.6.4

When I try to create repository I am getting 500 error more details below:

{"type":"amazon_client_exception","reason":"Unable to calculate a request signature: Unable to calculate a request signature: Algorithm HmacSHA1 not available","caused_by":{"type":"amazon_client_exception","reason":"Unable to calculate a request signature: Algorithm HmacSHA1 not available","caused_by":{"type":"no_such_algorithm_exception","reason":"Algorithm HmacSHA1 not available"}}}},"status":500

[dadoonet (David Pilato)]

This might indicate a broken installation of the JDK.

What is java -version printing?

[mmarinm (Marin Mestrovic)]

Hi, David thank you for your time and willingness to help.

And here is my Docker file for elastic:

FROM openjdk:8u121-jre-alpine

LABEL maintainer "itzg"

RUN apk -U add bash

ARG ES_VERSION=5.6.4
# avoid conflicts with debian host systems when mounting to host volume
ARG DEFAULT_ES_USER_UID=1100

ADD https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-$ES_VERSION.tar.gz /tmp
# need to adapt to both Docker's new remote-unpack-ADD behavior and the old behavior
RUN cd /usr/share && \
  if [ -f /tmp/elasticsearch-$ES_VERSION.tar.gz ]; then \
    tar xf /tmp/elasticsearch-$ES_VERSION.tar.gz; \
    else mv /tmp/elasticsearch-${ES_VERSION} /usr/share; \
    fi && \
  rm -f /tmp/elasticsearch-$ES_VERSION.tar.gz

EXPOSE 9200 9300

HEALTHCHECK --timeout=5s CMD wget -q -O - http://$HOSTNAME:9200/_cat/health

ENV ES_HOME=/usr/share/elasticsearch-$ES_VERSION \
    DEFAULT_ES_USER=elasticsearch \
    DEFAULT_ES_USER_UID=$DEFAULT_ES_USER_UID \
    ES_JAVA_OPTS="-Xms1g -Xmx1g"

RUN adduser -S -s /bin/sh -u $DEFAULT_ES_USER_UID $DEFAULT_ES_USER

VOLUME ["/data","/conf"]

WORKDIR $ES_HOME

COPY java.policy /usr/lib/jvm/java-1.8-openjdk/jre/lib/security/
COPY start /start
COPY log4j2.properties $ES_HOME/config/

CMD ["/start"]

[dadoonet (David Pilato)]

What do you have in java.policy, start and log4j2.properties?

Why not using the official elasticsearch docker image?

https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html

[mmarinm (Marin Mestrovic)]

this is a pretty much repo we are using U can see java.policy and log4j2.properties there.

I wish I can just switch to the official elastic image but that's not the case at the moment. This wasn't my choice but I have to live with it for now.

I changed java.policy in docker file to:

COPY java.policy /usr/lib/jvm/java-1.10.69-openjdk/jre/lib/security/

Since that is the version that plugin is using so I thought it might magically fix the issue, but it didn't. Now I am getting different error

[mmarinm (Marin Mestrovic)]

Issue solved, feel free to close

[dadoonet (David Pilato)]

COuld you share what the problem was and how you solved it?

[mmarinm (Marin Mestrovic)]

As I wrote earlier I don't know Java so I can't really identify the real issue. I solved it by trial, error and elimination.

• The first issue I had was with HmacSHA1 signature from the first post.

• My elastic container was working fine before I introduced repository s3. That's when I started to have this issue.

• Once I asked you for help you told me it might be broken installation of JDK, so I figure out, ok it must be that plugin installs something that breaks Java

• At this point, I entered the container and checked what does plugin install exactly. I figured out Java security policy plugin installs is version 1.10.69 and the version elasticsearch is using is 1.8

• So I thought ok that might be an issue and then I changed my docker file and I copied security policy for 1.10.69 and 1.8 which changed the error to MBeanServerPermission

• I tried to do curl request to take a snapshot and it succeeded despite the error in the elastic logs which I found weird

• I did some googling on MBeanServerPermission error and through Github issues, I found that I have to add additional permissions so I added in both policies 1.8 and 1.10.6:

    // JMX Java Management eXtensions
    permission javax.management.MBeanTrustPermission "register";
    permission javax.management.MBeanServerPermission "createMBeanServer";
    permission javax.management.MBeanPermission "-#-[-]", "queryNames";
    permission javax.management.MBeanServerPermission "findMBeanServer";
    permission javax.management.MBeanPermission "com.amazonaws.metrics.*", "*";
  

• that fixed the errors

Maybe you will be able to explain me why it worked