Hi,
I'm currently deploying the full stack on docker (each element is in a different container).
As a first test, I ran one container of each, and it worked well.
Using named volume I also add persistence to keep logstash's sincedb files and ES data.
Now, to do things properly, I'd like to deploy a cluster with multiple nodes. I've decided to have a master container (no data) and a data container, which can be scaled. My docker-compose (I omit logstash, kibana and other data volumes), looks like this:
docker-compose:
version: '2'
services:
elasticsearch_master:
build: elasticsearch/
volumes_from:
- data
ports:
- "9200:9200"
- "9300:9300"
environment:
bootstrap.memory_lock: "true"
ES_JAVA_OPTS: "-Xmx512m -Xms512m"
discovery.type: zen
node.master: "true"
node.data: "false"
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping_timeout: "5s"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
mem_limit: 1g
privileged: true
cap_add:
- IPC_LOCK
volumes:
- data_elasticsearch_master:/usr/share/elasticsearch/data
networks:
- elk
depends_on:
- data
elasticsearch:
build: elasticsearch/
volumes_from:
- data
ports:
- "9200"
- "9300"
links:
- elasticsearch_master
environment:
bootstrap.memory_lock: "true"
ES_JAVA_OPTS: "-Xmx2g -Xms2g"
discovery.type: zen
discovery.zen.ping.unicast.hosts: "elasticsearch_master"
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping_timeout: "5s"
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
mem_limit: 4g
privileged: true
cap_add:
- IPC_LOCK
volumes:
- data_elasticsearch:/usr/share/elasticsearch/data
networks:
- elk
depends_on:
- data
networks:
elk:
driver: bridge
volumes:
data_elasticsearch_master:
data_elasticsearch:
And to start it up, I'm doing: sudo docker-compose up -d --scale elasticsearch=2
So technically, I will have a 3 node cluster (1 master-only, 2 data).
The Dockerfile used to build images is the following:
Dockerfile:
# https://github.com/elastic/elasticsearch-docker
FROM docker.elastic.co/elasticsearch/elasticsearch:5.3.0
RUN rm -rf plugins/x-pack
USER root
RUN mkdir -p /usr/share/elasticsearch/data/
RUN chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/data/
USER elasticsearch
RUN chown -R elasticsearch:elasticsearch /usr/share/elasticsearch/
VOLUME /usr/share/elasticsearch/config
# Add your elasticsearch plugins setup here
# Example: RUN elasticsearch-plugin install analysis-icu
But a few moment later, I got this error from at least one ES data node:
[2017-06-27T08:16:26,516][INFO ][o.e.n.Node ] [] initializing ...
[2017-06-27T08:16:26,881][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data/docker-cluster]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:127) ~[elasticsearch-5.3.0.jar:5.3.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) ~[elasticsearch-5.3.0.jar:5.3.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:58) ~[elasticsearch-5.3.0.jar:5.3.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) ~[elasticsearch-5.3.0.jar:5.3.0]
at org.elasticsearch.cli.Command.main(Command.java:88) ~[elasticsearch-5.3.0.jar:5.3.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) ~[elasticsearch-5.3.0.jar:5.3.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) ~[elasticsearch-5.3.0.jar:5.3.0]
Caused by: java.lang.IllegalStateException: failed to obtain node locks, tried [[/usr/share/elasticsearch/data/docker-cluster]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.env.NodeEnvironment.<init>(NodeEnvironment.java:260) ~[elasticsearch-5.3.0.jar:5.3.0]
at org.elasticsearch.node.Node.<init>(Node.java:258) ~[elasticsearch-5.3.0.jar:5.3.0]
at org.elasticsearch.node.Node.<init>(Node.java:238) ~[elasticsearch-5.3.0.jar:5.3.0]
at org.elasticsearch.bootstrap.Bootstrap$6.<init>(Bootstrap.java:242) ~[elasticsearch-5.3.0.jar:5.3.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:242) ~[elasticsearch-5.3.0.jar:5.3.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:360) ~[elasticsearch-5.3.0.jar:5.3.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) ~[elasticsearch-5.3.0.jar:5.3.0]
... 6 more
2017-06-27 08:16:26,974 Thread-4 ERROR Unable to unregister MBeans java.security.AccessControlException: access denied ("javax.management.MBeanServerPermission" "createMBeanServer")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
at java.security.AccessController.checkPermission(AccessController.java:884)
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
at java.lang.management.ManagementFactory.getPlatformMBeanServer(ManagementFactory.java:465)
at org.apache.logging.log4j.core.jmx.Server.unregisterLoggerContext(Server.java:246)
at org.apache.logging.log4j.core.LoggerContext.stop(LoggerContext.java:317)
at org.apache.logging.log4j.core.AbstractLifeCycle.stop(AbstractLifeCycle.java:127)
at org.apache.logging.log4j.core.config.Configurator.shutdown(Configurator.java:344)
at org.elasticsearch.bootstrap.Bootstrap$5.run(Bootstrap.java:220)
I kinda get it that the problem is due to the fact that there is at least 2 nodes trying to store their data into the same directory.
I've read that node.max_locale_storage_node is not very recommended on production (and also to have different path for master/data but for the current problem, I don't think that's relevent), as it is recommended to only have one node per server.
But here I'm on containers, so I don't really know how to handle this.
Shall I increase this storage node? Or shall I do something else to allow "just in time" scaling? Or swarm is more compatible than compose?
Thanks.