Here is the error in elasticsearch logs when docker image build with 2.3.1.
[2016-04-12 14:30:00,083][WARN ][rest.suppressed ] /_cluster/health Params: {pretty=true}
MasterNotDiscoveredException[null]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$5.onTimeout(TransportMasterNodeAction.java:226)
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:236)
at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:804)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-04-12 14:30:51,440][WARN ][rest.suppressed ] /_cluster/health Params: {pretty=true}
MasterNotDiscoveredException[null]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$5.onTimeout(TransportMasterNodeAction.java:226)
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:236)
at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:804)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Config File: elasticsearch.yml
cluster.name: ${ES_ENV_CLUSTER_NAME}
node.name: ${HOSTNAME}
path.conf: ${ES_ENV_PATH_CONF}
path.data: ${ES_ENV_PATH_DATA}
path.work: ${ES_ENV_PATH_WORK}
path.logs: ${ES_ENV_PATH_LOGS}
path.plugins: ${ES_ENV_HOME}/plugins
bootstrap.mlockall: true
http.port: 9200
node.master: true
node.data: true
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: [ "${ES_ENV_M1}:9300","${ES_ENV_M2}:9300","${ES_ENV_M3}:9300" ]
indices.memory.index_buffer_size: 50%
discovery.zen.fd.ping_timeout: 30s
indices.fielddata.cache.size: 25%
action.disable_delete_all_indices: true
indices.cluster.send_refresh_mapping: false
cluster.routing.allocation.cluster_concurrent_rebalance: 2
cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: .90
cluster.routing.allocation.disk.watermark.high: .95
cluster.routing.allocation.node_concurrent_recoveries: 4
cluster.routing.allocation.node_initial_primaries_recoveries: 10
indices.store.throttle.max_bytes_per_sec: 200mb
threadpool.index.queue_size: 500
threadpool.bulk.queue_size: 500
http.max_content_length: 500mb
network.host: 0.0.0.0
Environment:
ES_ENV_PATH_LOGS=/es/logs
HOSTNAME=es03
ES_ENV_HOME=/usr/local/elasticsearch
ES_ENV_PATH_WORK=/es/work
ES_ENV_M1=es01
ES_ENV_M2=es02
ES_ENV_M3=es03
JAVA_HOME=/usr/lib/jvm/java-8-oracle
ES_ENV_PATH_PLUGINS=/usr/local/elasticsearch/plugins
ES_ENV_CONFIG=/es/config/elasticsearch.yml
ES_ENV_PATH_DATA=/es/data
ES_ENV_PATH_CONF=/es/config
ES_ENV_CLUSTER_NAME=es_test
Cluster Health when launched with 1.7.3:
elasticsearch@es03:/es/config$ curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "es_test",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0
}
Do you have three containers (one for each master) under the same VM? Or a single container running three ES nodes?
If three separate containers are involved then have you tried the setting
network.publish_host=$(docker-machine ip default)
They are on 3 different VMs. Network.host setting has been used with all interfaces as 0.0.0.0 so does it still need publish host setting?
ThanksAjayFrom: abhijatSent: Tuesday, April 12, 2016 2:13 PMTo: ajay.bh111@gmail.comReply To: Discuss Elasticsearch Logstash and Kibana | ElasticSubject: [Elasticsearch] Elasticsearch version 2.x can not start in docker container but 1.7.3 starts with same config
abhijat
April 12
ajay_bh111:
discovery.zen.ping.unicast.hosts: [ "${ES_ENV_M1}:9300","${ES_ENV_M2}:9300","${ES_ENV_M3}:9300" ]
Do you have three containers (one for each master) under the same VM? Or a single container running three ES nodes?
If three separate containers are involved then have you tried the setting
network.publish_host=$(docker-machine ip default)
Visit Topic or reply to this email to respond
To stop receiving notifications for this particular topic, click here. To unsubscribe from these emails, change your user preferences
Yes, I believe, you need to specifically set network.publish_host to the IP of the VM for other nodes to see it. I think there was a regression that got introduced with ES 2.2.0 around this and was subsequently fixed in 2.3.0. Try it.
Setting network.publish_host: cali0:ipv4 did not help with 2.3.0 but
worked with elasticsearch 1.7.3 (which worked without it too).
What is the IP address used for ES_ENV_M1, ES_ENV_M3 and ES_ENV_M3? It should be the IP address of the VMs on which the ES containers are running...
Also, cali0, should be the NIC of the VM on which the container is running.
These are docker containers with networking and ports 9200, 9300 are exposed between vms.
IP addresses are allocated by calico networking for docker containers. Container nodes IP can telnet to other containers on ports 9200 and 9300.
ThanksAjayFrom: abhijatSent: Wednesday, April 13, 2016 1:39 PMTo: ajay.bh111@gmail.comReply To: Discuss Elasticsearch Logstash and Kibana | ElasticSubject: [Elasticsearch] Elasticsearch version 2.x can not start in docker container but 1.7.3 starts with same config
abhijat
April 13ajay_bh111:
discovery.zen.ping.unicast.hosts: [ "${ES_ENV_M1}:9300","${ES_ENV_M2}:9300","${ES_ENV_M3}:9300" ]
What is the IP address used for ES_ENV_M1, ES_ENV_M3 and ES_ENV_M3? It should be the IP address of the VMs on which the ES containers are running...
Also, cali0, should be the NIC of the VM on which the container is running.
Visit Topic or reply to this email to respond
To stop receiving notifications for this particular topic, click here. To unsubscribe from these emails, change your user preferences
Here is what I have done:
I have two VMs created on my Mac. One with IP address 1.2.3.1 and the other with 1.2.3.2.
elasticsearch.yml:
cluster.name: es-local
network.host: ["_site:ipv4_", "_local:ipv4_"]
http.port: 9200
http.publish_port: 9200
http.compression: true
http.cors.enabled: true
http.cors.allow-credentials: true
http.cors.allow-origin: /.*/
transport.tcp.compress: true
transport.tcp.port: 9300
transport.publish_port: 9300
discovery.zen.ping_timeout: 10s
discovery.zen.join_timeout: 60s
script.groovy.sandbox.enabled: true
discovery.zen.ping.unicast.hosts: [1.2.3.1:9300]
On VM1 (1.2.3.1) and VM2 (1.2.3.2), I started ES node with above YML file using the command:
docker run -d -it -p 9200:9200 -p 9300:9300 \
-v $HOME/elasticsearch/standalone/data:/var/lib/elasticsearch \
-v $HOME/elasticsearch/standalone/log:/var/log/elasticsearch \
$MY_ES2X_DOCKER_IMAGE
And both ES nodes are able to connect to each other.
Whatever you use, you need to make sure that the IP address and ports for the VMs are properly set for discovery.zen.ping.unicast.hosts and that those are visible to other VMs and their corresponding containers.
Network connectivity is not issue. Another difference VMs are on different physical hypervisors. Probably ES is not handling multiple network interfaces on the containers correctly.
From: abhijatSent: Wednesday, April 13, 2016 2:40 PMTo: ajay.bh111@gmail.comReply To: Discuss Elasticsearch Logstash and Kibana | ElasticSubject: [Elasticsearch] Elasticsearch version 2.x can not start in docker container but 1.7.3 starts with same config
abhijat
April 13
Whatever you use, you need to make sure that the IP address and ports for the VMs are properly set for discovery.zen.ping.unicast.hosts and that those are visible to other VMs and their corresponding containers.
Visit Topic or reply to this email to respond
In Reply To
abhijat
April 13Here is what I have done:
I have two VMs created on my Mac. One with IP address 1.2.3.1 and the other with 1.2.3.2.
Elasticsearch.yml:
cluster.name: es-local
network.host: ["site:ipv4", "local:ipv4"]
http.port: 9200
http.publish_port: 9200
http.compression: true
http.cors.enabled: true
ht…
Visit Topic or reply to this email to respond
To stop receiving notifications for this particular topic, click here. To unsubscribe from these emails, change your user preferences
It shouldnt matter IMO. In your case you have to specify the IP address of the machines on which the containers are executing. Say, you have two physically separate boxes host1 and host2, then you need to put the IP address and port of the host1 and host2 in your YML files.
Let us take a simple case... have you tried starting only a single instance of ES 2.3 on one of these boxes and tried to access via the browser? something like http://host1:9200/? And if it works then what value do you specify for host1?