Elasticsearch version 2.x can not start in docker container but 1.7.3 starts with same config

ajay_bh111 · April 12, 2016, 3:38pm

Here is the error in elasticsearch logs when docker image build with 2.3.1.

[2016-04-12 14:30:00,083][WARN ][rest.suppressed ] /_cluster/health Params: {pretty=true}
MasterNotDiscoveredException[null]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$5.onTimeout(TransportMasterNodeAction.java:226)
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:236)
at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:804)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-04-12 14:30:51,440][WARN ][rest.suppressed ] /_cluster/health Params: {pretty=true}
MasterNotDiscoveredException[null]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$5.onTimeout(TransportMasterNodeAction.java:226)
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:236)
at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:804)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Config File: elasticsearch.yml
cluster.name: ${ES_ENV_CLUSTER_NAME}
node.name: ${HOSTNAME}
path.conf: ${ES_ENV_PATH_CONF}
path.data: ${ES_ENV_PATH_DATA}
path.work: ${ES_ENV_PATH_WORK}
path.logs: ${ES_ENV_PATH_LOGS}
path.plugins: ${ES_ENV_HOME}/plugins
bootstrap.mlockall: true
http.port: 9200
node.master: true
node.data: true
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: [ "${ES_ENV_M1}:9300","${ES_ENV_M2}:9300","${ES_ENV_M3}:9300" ]
indices.memory.index_buffer_size: 50%
discovery.zen.fd.ping_timeout: 30s
indices.fielddata.cache.size: 25%
action.disable_delete_all_indices: true
indices.cluster.send_refresh_mapping: false
cluster.routing.allocation.cluster_concurrent_rebalance: 2
cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: .90
cluster.routing.allocation.disk.watermark.high: .95
cluster.routing.allocation.node_concurrent_recoveries: 4
cluster.routing.allocation.node_initial_primaries_recoveries: 10
indices.store.throttle.max_bytes_per_sec: 200mb
threadpool.index.queue_size: 500
threadpool.bulk.queue_size: 500
http.max_content_length: 500mb
network.host: 0.0.0.0

Environment:
ES_ENV_PATH_LOGS=/es/logs
HOSTNAME=es03
ES_ENV_HOME=/usr/local/elasticsearch
ES_ENV_PATH_WORK=/es/work
ES_ENV_M1=es01
ES_ENV_M2=es02
ES_ENV_M3=es03
JAVA_HOME=/usr/lib/jvm/java-8-oracle
ES_ENV_PATH_PLUGINS=/usr/local/elasticsearch/plugins
ES_ENV_CONFIG=/es/config/elasticsearch.yml
ES_ENV_PATH_DATA=/es/data
ES_ENV_PATH_CONF=/es/config
ES_ENV_CLUSTER_NAME=es_test

Cluster Health when launched with 1.7.3:
elasticsearch@es03:/es/config$ curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "es_test",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0
}

abhijat · April 12, 2016, 6:05pm

Do you have three containers (one for each master) under the same VM? Or a single container running three ES nodes?

If three separate containers are involved then have you tried the setting

network.publish_host=$(docker-machine ip default)

ajay_bh111 · April 12, 2016, 6:28pm

They are on 3 different VMs. Network.host setting has been used with all interfaces as 0.0.0.0 so does it still need publish host setting?

ThanksAjayFrom: abhijatSent: Tuesday, April 12, 2016 2:13 PMTo: ajay.bh111@gmail.comReply To: Discuss Elasticsearch Logstash and Kibana | ElasticSubject: [Elasticsearch] Elasticsearch version 2.x can not start in docker container but 1.7.3 starts with same config

abhijat

April 12ajay_bh111:

discovery.zen.ping.unicast.hosts: [ "${ES_ENV_M1}:9300","${ES_ENV_M2}:9300","${ES_ENV_M3}:9300" ]

Do you have three containers (one for each master) under the same VM? Or a single container running three ES nodes?

If three separate containers are involved then have you tried the setting

network.publish_host=$(docker-machine ip default)

Visit Topic or reply to this email to respond

To stop receiving notifications for this particular topic, click here. To unsubscribe from these emails, change your user preferences

abhijat · April 12, 2016, 7:23pm

Yes, I believe, you need to specifically set network.publish_host to the IP of the VM for other nodes to see it. I think there was a regression that got introduced with ES 2.2.0 around this and was subsequently fixed in 2.3.0. Try it.

ajay_bh111 · April 13, 2016, 2:45pm

Setting network.publish_host: cali0:ipv4 did not help with 2.3.0 but
worked with elasticsearch 1.7.3 (which worked without it too).

abhijat · April 13, 2016, 5:29pm

What is the IP address used for ES_ENV_M1, ES_ENV_M3 and ES_ENV_M3? It should be the IP address of the VMs on which the ES containers are running...

Also, cali0, should be the NIC of the VM on which the container is running.

ajay_bh111 · April 13, 2016, 5:44pm

These are docker containers with networking and ports 9200, 9300 are exposed between vms.

IP addresses are allocated by calico networking for docker containers. Container nodes IP can telnet to other containers on ports 9200 and 9300.

ThanksAjayFrom: abhijatSent: Wednesday, April 13, 2016 1:39 PMTo: ajay.bh111@gmail.comReply To: Discuss Elasticsearch Logstash and Kibana | ElasticSubject: [Elasticsearch] Elasticsearch version 2.x can not start in docker container but 1.7.3 starts with same config

abhijat

April 13ajay_bh111:

discovery.zen.ping.unicast.hosts: [ "${ES_ENV_M1}:9300","${ES_ENV_M2}:9300","${ES_ENV_M3}:9300" ]

What is the IP address used for ES_ENV_M1, ES_ENV_M3 and ES_ENV_M3? It should be the IP address of the VMs on which the ES containers are running...

Also, cali0, should be the NIC of the VM on which the container is running.

Visit Topic or reply to this email to respond

To stop receiving notifications for this particular topic, click here. To unsubscribe from these emails, change your user preferences

abhijat · April 13, 2016, 6:06pm

Here is what I have done:

I have two VMs created on my Mac. One with IP address 1.2.3.1 and the other with 1.2.3.2.

elasticsearch.yml:

cluster.name: es-local
network.host: ["_site:ipv4_", "_local:ipv4_"]

http.port: 9200
http.publish_port: 9200
http.compression: true
http.cors.enabled: true
http.cors.allow-credentials: true
http.cors.allow-origin: /.*/

transport.tcp.compress: true
transport.tcp.port: 9300
transport.publish_port: 9300

discovery.zen.ping_timeout: 10s
discovery.zen.join_timeout: 60s

script.groovy.sandbox.enabled: true

discovery.zen.ping.unicast.hosts: [1.2.3.1:9300]

On VM1 (1.2.3.1) and VM2 (1.2.3.2), I started ES node with above YML file using the command:

docker run -d -it -p 9200:9200 -p 9300:9300 \
    -v $HOME/elasticsearch/standalone/data:/var/lib/elasticsearch \
    -v $HOME/elasticsearch/standalone/log:/var/log/elasticsearch \
$MY_ES2X_DOCKER_IMAGE

And both ES nodes are able to connect to each other.

abhijat · April 13, 2016, 6:30pm

Whatever you use, you need to make sure that the IP address and ports for the VMs are properly set for discovery.zen.ping.unicast.hosts and that those are visible to other VMs and their corresponding containers.

ajay_bh111 · April 13, 2016, 6:48pm

Network connectivity is not issue. Another difference VMs are on different physical hypervisors. Probably ES is not handling multiple network interfaces on the containers correctly.

From: abhijatSent: Wednesday, April 13, 2016 2:40 PMTo: ajay.bh111@gmail.comReply To: Discuss Elasticsearch Logstash and Kibana | ElasticSubject: [Elasticsearch] Elasticsearch version 2.x can not start in docker container but 1.7.3 starts with same config

abhijat

April 13

Whatever you use, you need to make sure that the IP address and ports for the VMs are properly set for discovery.zen.ping.unicast.hosts and that those are visible to other VMs and their corresponding containers.

Visit Topic or reply to this email to respond

In Reply To

abhijat

April 13Here is what I have done:

I have two VMs created on my Mac. One with IP address 1.2.3.1 and the other with 1.2.3.2.

elasticsearch.yml:

cluster.name: es-local
network.host: ["site:ipv4", "local:ipv4"]

http.port: 9200
http.publish_port: 9200
http.compression: true
http.cors.enabled: true
ht…

Visit Topic or reply to this email to respond

To stop receiving notifications for this particular topic, click here. To unsubscribe from these emails, change your user preferences

abhijat · April 13, 2016, 7:10pm

It shouldnt matter IMO. In your case you have to specify the IP address of the machines on which the containers are executing. Say, you have two physically separate boxes host1 and host2, then you need to put the IP address and port of the host1 and host2 in your YML files.

Let us take a simple case... have you tried starting only a single instance of ES 2.3 on one of these boxes and tried to access via the browser? something like http://host1:9200/? And if it works then what value do you specify for host1?

system · July 5, 2017, 10:59pm