# Cannot start Elasticsearch when using repository-s3 plugin **URL:** \ **Category:** Elasticsearch\ **Created:** [August 8, 2017, 5:55pm UTC](https://discuss.elastic.co/t/cannot-start-elasticsearch-when-using-repository-s3-plugin/96318 "2017-08-08T17:55:45Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![ps\_tom](https://avatars.discourse-cdn.com/v4/letter/p/d07c76/32.png) [@ps\_tom](https://discuss.elastic.co/u/ps_tom)\ **Post date:** [August 8, 2017, 5:55pm UTC](https://discuss.elastic.co/t/cannot-start-elasticsearch-when-using-repository-s3-plugin/96318/1 "2017-08-08T17:55:45Z")
I am using ES 5.5.1 running within Docker containers spread across 3 AWS EC2 instances. These are run based on the official Elastic Docker image from [docker.elastic.co/elasticsearch/elasticsearch:5.5.1](http://docker.elastic.co/elasticsearch/elasticsearch:5.5.1), with the repository-s3 plugin being installed as a step within my Dockerfile. However, when I try to start my nodes with this plugin installed, they seem to throw a variety of exceptions. If I start the nodes using the exact same config but without the repository-s3 plugin, then everything starts as expected. The contents of one of the docker-compose files is as follows: ``` version: '2' services: elasticsearch1: image: docker.elastic.co/elasticsearch/elasticsearch:5.5.1 container_name: elasticsearch1 restart: always environment: - cluster.name=docker-cluster - node.master=true - bootstrap.memory_lock=true - "ES_JAVA_OPTS=-Xms1g -Xmx1g -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses" - "network.publish_host=172.31.32.168:9300" - "discovery.zen.ping.unicast.hosts=172.31.32.168:9300,172.31.104.72:9300,172.31.167.200:9300" - "discovery.zen.ping_timeout=5m" - "discovery.zen.minimum_master_nodes=2" - "network.bind_host=0.0.0.0" - "xpack.security.enabled=false" - "xpack.ml.enabled=false" - "xpack.watcher.enabled=false" ulimits: memlock: soft: -1 hard: -1 mem_limit: 2g volumes: - esdata1:/usr/share/elasticsearch/data ports: - 9200:9200 - 9300:9300 volumes: esdata1: driver: local ``` And the contents of the Dockerfile is as follows: ``` FROM docker.elastic.co/elasticsearch/elasticsearch:5.5.1 USER elasticsearch RUN elasticsearch-plugin install --batch repository-s3 && \ bin/elasticsearch-keystore create && \ echo ************ | bin/elasticsearch-keystore add -stdin cloud.aws.s3.access_key && \ echo ************ | bin/elasticsearch-keystore add -stdin cloud.aws.s3.secret_key ``` Hence the only difference between the 2 setups is that the s3-repository has been installed, along with the keys being added to the keystore. The logs can be found here (wouldn't let me put them here due to character limit): [https://pastebin.com/mQnFfZ3F](https://pastebin.com/mQnFfZ3F) NB - The discovery is using the standard zen unicast discovery as I couldn't seem to get the EC2 discovery plugin working either. ---
**Author:** ![rjernst](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rjernst/32/6363_2.png) [@rjernst](https://discuss.elastic.co/u/rjernst)\ **Post date:** [August 9, 2017, 2:56am UTC](https://discuss.elastic.co/t/cannot-start-elasticsearch-when-using-repository-s3-plugin/96318/2 "2017-08-09T02:56:25Z")
The settings you are putting in the keystore have the wrong name. They should be `s3.client.default.access_key` and `s3.client.default.secret_key`. ---
**Author:** ![ps\_tom](https://avatars.discourse-cdn.com/v4/letter/p/d07c76/32.png) [@ps\_tom](https://discuss.elastic.co/u/ps_tom)\ **Post date:** [August 9, 2017, 8:33am UTC](https://discuss.elastic.co/t/cannot-start-elasticsearch-when-using-repository-s3-plugin/96318/3 "2017-08-09T08:33:22Z")
Even with these settings corrected, it still throws up exceptions when attempting to start the cluster? ---
**Author:** ![ps\_tom](https://avatars.discourse-cdn.com/v4/letter/p/d07c76/32.png) [@ps\_tom](https://discuss.elastic.co/u/ps_tom)\ **Post date:** [August 10, 2017, 8:43am UTC](https://discuss.elastic.co/t/cannot-start-elasticsearch-when-using-repository-s3-plugin/96318/4 "2017-08-10T08:43:41Z")
Anyone got any ideas on this one? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [August 10, 2017, 9:10am UTC](https://discuss.elastic.co/t/cannot-start-elasticsearch-when-using-repository-s3-plugin/96318/5 "2017-08-10T09:10:35Z")
> it still throws up exceptions Exactly the same exceptions? ---
**Author:** ![ps\_tom](https://avatars.discourse-cdn.com/v4/letter/p/d07c76/32.png) [@ps\_tom](https://discuss.elastic.co/u/ps_tom)\ **Post date:** [August 10, 2017, 10:11am UTC](https://discuss.elastic.co/t/cannot-start-elasticsearch-when-using-repository-s3-plugin/96318/6 "2017-08-10T10:11:07Z")
Just tried again to double check on the exceptions. Appears that nothing can connect to the master node? Again, if I start this up without the s3-repository installed, everything works as expected. The logs are here: [https://pastebin.com/kJr1JFKL](https://pastebin.com/kJr1JFKL) ---
**Author:** ![baz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/baz/32/5183_2.png) [@baz](https://discuss.elastic.co/u/baz)\ **Post date:** [August 15, 2017, 3:46pm UTC](https://discuss.elastic.co/t/cannot-start-elasticsearch-when-using-repository-s3-plugin/96318/7 "2017-08-15T15:46:53Z")
I am currently trying to run a 3 node scenario on a single host to validate what you are saying @ps_tom , but one thing i noticed immediately is that the `publish_host` setting you've set is incorrect. It should _not_ have the port, and only be `- "network.publish_host=172.31.32.XXX"`. Just for completeness, it is fine that the `discovery.zen.ping.unicast.hosts` has `host:port` combinations, so you need not change those. Try it out w/ the proper `publish_host` and see what happens. Ill post a 3 node working config here too, soon. ---
**Author:** ![baz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/baz/32/5183_2.png) [@baz](https://discuss.elastic.co/u/baz)\ **Post date:** [August 17, 2017, 3:13pm UTC](https://discuss.elastic.co/t/cannot-start-elasticsearch-when-using-repository-s3-plugin/96318/8 "2017-08-17T15:13:40Z")
There was indeed an issue in 5.5. It was caused by serialization when the node stats for a node with secure settings (keystore settings) had a bad value read for its size. The fix is [here](https://github.com/elastic/elasticsearch/commit/6982531108f2ffb27b342a167da6ccfc992b2fc3). It will be fixed in 5.6 when it comes out. Another issue I see with the above snippet is the Dockerfile RUN line has a single dash for `-stdin` and it should be two. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 14, 2017, 3:14pm UTC](https://discuss.elastic.co/t/cannot-start-elasticsearch-when-using-repository-s3-plugin/96318/9 "2017-09-14T15:14:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.