Cannot start Elasticsearch when using repository-s3 plugin

Elastic Stack Elasticsearch
1

I am using ES 5.5.1 running within Docker containers spread across 3 AWS EC2 instances. These are run based on the official Elastic Docker image from docker.elastic.co/elasticsearch/elasticsearch:5.5.1, with the repository-s3 plugin being installed as a step within my Dockerfile. However, when I try to start my nodes with this plugin installed, they seem to throw a variety of exceptions. If I start the nodes using the exact same config but without the repository-s3 plugin, then everything starts as expected.

The contents of one of the docker-compose files is as follows:

version: '2'
services:
  elasticsearch1:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.5.1
    container_name: elasticsearch1
    restart: always
    environment:
      - cluster.name=docker-cluster
      - node.master=true
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms1g -Xmx1g -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses"
      - "network.publish_host=172.31.32.168:9300"
      - "discovery.zen.ping.unicast.hosts=172.31.32.168:9300,172.31.104.72:9300,172.31.167.200:9300"
      - "discovery.zen.ping_timeout=5m"
      - "discovery.zen.minimum_master_nodes=2"
      - "network.bind_host=0.0.0.0"
      - "xpack.security.enabled=false"
      - "xpack.ml.enabled=false"
      - "xpack.watcher.enabled=false"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 2g
    volumes:
      - esdata1:/usr/share/elasticsearch/data
    ports:
      - 9200:9200
      - 9300:9300

volumes:
  esdata1:
    driver: local

And the contents of the Dockerfile is as follows:

FROM docker.elastic.co/elasticsearch/elasticsearch:5.5.1

USER elasticsearch
RUN elasticsearch-plugin install --batch repository-s3 && \
    bin/elasticsearch-keystore create && \
    echo ************ | bin/elasticsearch-keystore add -stdin cloud.aws.s3.access_key && \
    echo ************ | bin/elasticsearch-keystore add -stdin cloud.aws.s3.secret_key

Hence the only difference between the 2 setups is that the s3-repository has been installed, along with the keys being added to the keystore.

The logs can be found here (wouldn't let me put them here due to character limit):
https://pastebin.com/mQnFfZ3F

NB - The discovery is using the standard zen unicast discovery as I couldn't seem to get the EC2 discovery plugin working either.

1 Like
2

The settings you are putting in the keystore have the wrong name. They should be s3.client.default.access_key and s3.client.default.secret_key.

3

Even with these settings corrected, it still throws up exceptions when attempting to start the cluster?

4

Anyone got any ideas on this one?

5

it still throws up exceptions

Exactly the same exceptions?

6

Just tried again to double check on the exceptions. Appears that nothing can connect to the master node? Again, if I start this up without the s3-repository installed, everything works as expected.

The logs are here:
https://pastebin.com/kJr1JFKL

7

I am currently trying to run a 3 node scenario on a single host to validate what you are saying @ps_tom , but one thing i noticed immediately is that the publish_host setting you've set is incorrect. It should not have the port, and only be - "network.publish_host=172.31.32.XXX". Just for completeness, it is fine that the discovery.zen.ping.unicast.hosts has host:port combinations, so you need not change those. Try it out w/ the proper publish_host and see what happens. Ill post a 3 node working config here too, soon.

8

There was indeed an issue in 5.5. It was caused by serialization when the node stats for a node with secure settings (keystore settings) had a bad value read for its size. The fix is here. It will be fixed in 5.6 when it comes out.

Another issue I see with the above snippet is the Dockerfile RUN line has a single dash for -stdin and it should be two.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.