# Different ES clusters in the same host **URL:** https://discuss.elastic.co/t/different-es-clusters-in-the-same-host/22006 **Category:** Elasticsearch **Created:** [February 6, 2015, 12:23am UTC](https://discuss.elastic.co/t/different-es-clusters-in-the-same-host/22006 "2015-02-06T00:23:35Z") **Posts on this page:** 4 **Page:** 1
### Author: ![Ricardo\_Alves\_dos\_Re](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ricardo_alves_dos_re/32/932_2.png) [@Ricardo\_Alves\_dos\_Re](https://discuss.elastic.co/u/Ricardo_Alves_dos_Re) #### Post date: [February 6, 2015, 12:23am UTC](https://discuss.elastic.co/t/different-es-clusters-in-the-same-host/22006/1 "2015-02-06T00:23:35Z")
Hello folks, First of all, I'd like to explain my current necessity to bring this future trend out, some projects in my company have bean developed to use a lot of micro-services, unfortunately, is not uncommon that the mean of cpu usage in our cloud servers (AWS) are sustained in a lower degrees. To provisioning these resources easily all the components are executed in a docker containers and these stacks were represented as yaml file ([http://www.fig.sh](http://www.fig.sh)). In a group of servers I've more than one Elasticsearch clusters, containerized to use EC2 discovery and with a transport.publish\_port configured to avoid collision, however, only the first cluster in default transport port have a correct cluster formation. My guess is the transport.publish\_post is not used as pointer to discovery.zen.unicast to avoid unused transport port.. Thanks for any kind of help. #docker #aws #discovery fig.yml esemail: image: XXX/elasticsearch:latest volumes: - /dev/log:/dev/log - /data0/email:/data/0 - /data1/email:/data/1 ports: - "9202:9200" - "9302:9300" environment: TRANSPORT\_PUBLISH\_PORT: 9302 NETWORK\_PUBLISH\_HOST: _ec2_ CLUSTER\_ROUTING\_ALLOCATION\_SAME\_SHARD\_HOST: "true" ES\_ROOT\_LOG\_LEVEL: INFO ES\_DISCOVERY\_LOG\_LEVEL: TRACE ES\_LOG\_DIR: /var/log/elasticsearch ES\_HEAP\_SIZE: 1g ES\_DIRECT\_SIZE: 256m ES\_JAVA\_OPTS: "-server -XX:+UseCompressedOops -XX:+DoEscapeAnalysis -XX:+AlwaysPreTouch -verbose:gc -Xloggc:/var/log/elasticsearch/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=512K" MAX\_OPEN\_FILES: 65535 MAX\_MAP\_COUNT: 262144 CLUSTER\_NAME: email-es NODE\_MASTER: "true" NODE\_DATA: "true" PATH\_DATA: /data/0/datasearch0,/data/1/datasearch0 BOOTSTRAP\_MLOCKALL: "false" DISCOVERY\_ZEN\_MCAST: "false" DISCOVERY\_ZEN\_REJOIN\_ON\_MASTER\_GONE: "false" DISCOVERY\_ZEN\_MINIMUM\_MASTER\_NODES: 2 DISCOVERY\_TYPE: ec2 DISCOVERY\_EC2\_ANY\_GROUP: "false" DISCOVERY\_EC2\_GROUPS: db-prod-es-mail DISCOVERY\_EC2\_AVAILABILITY\_ZONES: us-east-1a,us-east-1b,us-east-1d DISCOVERY\_EC2\_HOST\_TYPE: private\_ip DISCOVERY\_EC2\_PING\_TIMEOUT: 1m GATEWAY\_RECOVER\_AFTER\_NODES: 2 GATEWAY\_RECOVER\_AFTER\_TIME: 2m GATEWAY\_EXPECTED\_NODES: 3 AWS\_ELB\_NAME: ESEmailBionexo AWS\_ACCESS\_KEY\_ID: XXXXX AWS\_SECRET\_ACCESS\_KEY: XXXX AWS\_DEFAULT\_REGION: us-east-1 THREAD\_POOL\_SEARCH: 2000 THREAD\_POOL\_INDEX: 2000 THREAD\_POOL\_BULK: 2000 [ec2-user@ip-172-16-255-20 ~]$ sudo docker ps |grep elastic d1b9ebf8c605 bionexo/elasticsearch:2015020502 "/bin/sh -c /bionexo 3 hours ago Up 3 hours 0.0.0.0:9202-\>9200/tcp, 0.0.0.0:9302-\>9300/tcp elasticsearch\_esemail\_1 088da31f943b bionexo/elasticsearch:2015020401 "/bin/sh -c /bionexo 35 hours ago Up 35 hours 0.0.0.0:9200-\>9200/tcp, 0.0.0.0:9300-\>9300/tcp elasticsearch\_eslogger\_1 [2015-02-05 20:53:09,834][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:09,853][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:09,854][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:10,637][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:10,637][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:23,833][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:25,837][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:26,618][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:35,724][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:38,034][TRACE][discovery.ec2] [d1b9ebf8c605] building dynamic unicast discovery nodes... [2015-02-05 20:53:38,034][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-713a399d based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,034][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-7bd2f097 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,035][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-0b25abfa based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,035][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-83c39ca2 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: automation,GroupId: sg-90c510f5}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,035][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-7d49e897 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,035][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-fb9cc0d7 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,035][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-a72ef14d based on groups [{GroupName: corporativo,GroupId: sg-8f9c46ea}, {GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,035][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-54763907 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,035][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-b98e5e48 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,036][TRACE][discovery.ec2] [d1b9ebf8c605] adding i-b0384441, address 172.16.250.83, transport\_address inet[/172.16.250.83:9300] [2015-02-05 20:53:38,036][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-84b324d7 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: demo\_apps,GroupId: sg-44c31d21}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,036][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-eb29db01 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,037][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-6cdd259d based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,037][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-53cabead based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,037][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-f5f21004 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,037][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-fd21a317 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,037][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-ebd815bb based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,037][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-81187b70 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,038][TRACE][discovery.ec2] [d1b9ebf8c605] adding i-4d52afb7, address 172.16.255.20, transport\_address inet[/172.16.255.20:9300] [2015-02-05 20:53:38,039][TRACE][discovery.ec2] [d1b9ebf8c605] adding i-90f2367f, address 172.16.255.149, transport\_address inet[/172.16.255.149:9300] [2015-02-05 20:53:38,039][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-b25d8e5f based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,039][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-4517fab4 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,039][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-573bf607 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: vpn,GroupId: sg-4fac3e2a}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,039][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-ca09f627 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: varnish,GroupId: sg-3815eb5c}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,040][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-a2728b53 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,040][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-fb222600 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,040][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-502433bc based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,040][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-2e871305 based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,040][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-3a7f91c0 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,040][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-19a469e3 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,040][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-e8038e04 based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail] [2015-02-05 20:53:38,040][DEBUG][discovery.ec2] [d1b9ebf8c605] using dynamic discovery nodes [[#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]], [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]], [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]] [2015-02-05 20:53:38,041][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] replacing [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]] with temp node [#zen\_unicast\_22\_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]] [2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connecting (light) to [#zen\_unicast\_22\_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]] [2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] replacing [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]] with temp node [#zen\_unicast\_23\_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]] [2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connecting (light) to [#zen\_unicast\_23\_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]] [2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] replacing [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]] with temp node [#zen\_unicast\_24\_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]] [2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connected to [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]] [2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [#zen\_unicast\_22\_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]], using \>=1.4.0 serialization [2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connecting (light) to [#zen\_unicast\_24\_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]] [2015-02-05 20:53:38,044][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connected to [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]] [2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [#zen\_unicast\_23\_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]], using \>=1.4.0 serialization [2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connected to [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]] [2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [#zen\_unicast\_24\_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]], using \>=1.4.0 serialization [2015-02-05 20:53:38,046][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true} [2015-02-05 20:53:38,055][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [#zen\_unicast\_23\_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]: [ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[5], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true}], id[442], master [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], hasJoinedOnce [true], cluster\_name[logger]}] [2015-02-05 20:53:38,054][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [#zen\_unicast\_22\_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]: [ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[5], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], id[454], master [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], hasJoinedOnce [true], cluster\_name[logger]}] [2015-02-05 20:53:38,055][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] filtering out response from [088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}, not same cluster\_name [logger] [2015-02-05 20:53:38,056][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] filtering out response from [1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true}, not same cluster\_name [logger] [2015-02-05 20:53:38,057][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}: [ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[5], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[16], master [null], hasJoinedOnce [false], cluster\_name[email-es]}] [2015-02-05 20:53:38,857][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:38,864][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:38,864][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:38,866][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:38,886][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:38,890][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:39,659][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:39,660][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:39,665][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:39,674][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:39,714][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:39,719][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:39,722][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:39,744][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:39,851][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:39,855][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:39,854][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:39,853][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:39,857][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:39,859][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:39,927][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:39,929][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:39,929][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:39,931][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:39,931][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:39,934][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:39,934][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:39,935][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:39,937][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:39,937][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:39,938][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,037][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,038][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,039][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,044][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,044][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,046][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,047][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,048][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,048][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,049][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,049][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,050][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,053][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,054][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,068][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,071][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,074][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,079][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,080][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,082][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,619][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,622][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:40,623][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,632][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:40,632][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:40,634][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,636][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:40,644][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:40,647][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:40,657][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,658][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:40,662][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,671][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,671][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,671][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,675][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:40,684][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:40,686][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:40,687][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:40,687][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:40,689][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:40,689][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:40,690][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:40,691][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:40,692][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:40,692][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:40,692][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:40,694][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:40,692][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:40,695][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:40,694][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:40,700][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:40,700][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:40,985][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:41,172][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:41,177][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:41,182][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:41,183][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:41,186][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:41,188][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:41,194][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:41,197][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:41,200][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:41,201][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:41,203][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:41,203][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:41,207][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:41,208][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:41,210][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:41,211][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:41,215][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:41,218][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:41,229][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:41,231][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:41,234][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:41,236][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:41,240][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:41,241][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:41,241][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:41,242][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:42,249][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:43,617][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:44,206][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:45,041][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [#zen\_unicast\_24\_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]: [ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[5], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[22b068e7698f][HZKmb28dQRO7Zc4QR-\_qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true}], id[43], master [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], hasJoinedOnce [true], cluster\_name[logger]}] [2015-02-05 20:53:45,041][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] filtering out response from [22b068e7698f][HZKmb28dQRO7Zc4QR-\_qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true}, not same cluster\_name [logger] [2015-02-05 20:53:45,288][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:46,241][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:48,549][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:48,808][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:49,835][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:51,279][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:52,079][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:53,051][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:53,359][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:54,618][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:54,621][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:54,850][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:57,485][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:53:58,526][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:53:59,842][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:54:00,938][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:54:01,047][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:54:02,304][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:54:03,105][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:54:04,315][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:54:06,436][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:54:08,295][TRACE][discovery.ec2] [d1b9ebf8c605] building dynamic unicast discovery nodes... [2015-02-05 20:54:08,297][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-713a399d based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,297][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-7bd2f097 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,297][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-0b25abfa based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,297][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-83c39ca2 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: automation,GroupId: sg-90c510f5}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,297][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-7d49e897 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,298][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-fb9cc0d7 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,298][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-a72ef14d based on groups [{GroupName: corporativo,GroupId: sg-8f9c46ea}, {GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,298][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-54763907 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,299][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-b98e5e48 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,301][TRACE][discovery.ec2] [d1b9ebf8c605] adding i-b0384441, address 172.16.250.83, transport\_address inet[/172.16.250.83:9300] [2015-02-05 20:54:08,301][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-84b324d7 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: demo\_apps,GroupId: sg-44c31d21}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,301][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-eb29db01 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,301][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-6cdd259d based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,301][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-53cabead based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,302][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-f5f21004 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,302][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-fd21a317 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,302][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-ebd815bb based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,302][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-81187b70 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,304][TRACE][discovery.ec2] [d1b9ebf8c605] adding i-4d52afb7, address 172.16.255.20, transport\_address inet[/172.16.255.20:9300] [2015-02-05 20:54:08,312][TRACE][discovery.ec2] [d1b9ebf8c605] adding i-90f2367f, address 172.16.255.149, transport\_address inet[/172.16.255.149:9300] [2015-02-05 20:54:08,313][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-b25d8e5f based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,313][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-4517fab4 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,313][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-573bf607 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: vpn,GroupId: sg-4fac3e2a}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,315][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-ca09f627 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: varnish,GroupId: sg-3815eb5c}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,316][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-a2728b53 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,316][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-fb222600 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,316][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-502433bc based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,316][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-2e871305 based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,316][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-3a7f91c0 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,316][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-19a469e3 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,317][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-e8038e04 based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail] [2015-02-05 20:54:08,317][DEBUG][discovery.ec2] [d1b9ebf8c605] using dynamic discovery nodes [[#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]], [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]], [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]] [2015-02-05 20:54:08,317][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] replacing [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]] with temp node [#zen\_unicast\_25\_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]] [2015-02-05 20:54:08,317][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connecting (light) to [#zen\_unicast\_25\_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]] [2015-02-05 20:54:08,318][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] replacing [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]] with temp node [#zen\_unicast\_26\_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]] [2015-02-05 20:54:08,319][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connected to [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]] [2015-02-05 20:54:08,319][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [#zen\_unicast\_25\_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]], using \>=1.4.0 serialization [2015-02-05 20:54:08,322][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [#zen\_unicast\_25\_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]: [ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], id[457], master [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], hasJoinedOnce [true], cluster\_name[logger]}] [2015-02-05 20:54:08,322][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] filtering out response from [088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}, not same cluster\_name [logger] [2015-02-05 20:54:08,325][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connecting (light) to [#zen\_unicast\_26\_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]] [2015-02-05 20:54:08,326][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] replacing [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]] with temp node [#zen\_unicast\_27\_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]] [2015-02-05 20:54:08,327][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connecting (light) to [#zen\_unicast\_27\_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]] [2015-02-05 20:54:08,327][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true} [2015-02-05 20:54:08,328][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connected to [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]] [2015-02-05 20:54:08,334][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [#zen\_unicast\_27\_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]], using \>=1.4.0 serialization [2015-02-05 20:54:08,328][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connected to [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]] [2015-02-05 20:54:08,334][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [#zen\_unicast\_26\_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]], using \>=1.4.0 serialization [2015-02-05 20:54:08,341][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [#zen\_unicast\_26\_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]: [ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true}], id[445], master [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], hasJoinedOnce [true], cluster\_name[logger]}] [2015-02-05 20:54:08,341][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] filtering out response from [1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true}, not same cluster\_name [logger] [2015-02-05 20:54:08,342][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}: [ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[18], master [null], hasJoinedOnce [false], cluster\_name[email-es]}] [2015-02-05 20:54:09,814][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:54:09,835][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:54:10,616][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:54:10,637][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:54:10,638][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] [2015-02-05 20:54:10,854][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:54:10,857][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry [2015-02-05 20:54:11,404][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [#zen\_unicast\_27\_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]: [ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[22b068e7698f][HZKmb28dQRO7Zc4QR-\_qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true}], id[46], master [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], hasJoinedOnce [true], cluster\_name[logger]}] [2015-02-05 20:54:11,405][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] filtering out response from [22b068e7698f][HZKmb28dQRO7Zc4QR-\_qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true}, not same cluster\_name [logger] [2015-02-05 20:54:11,405][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen\_unicast\_19\_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]] [2015-02-05 20:54:11,406][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen\_unicast\_22\_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]] [2015-02-05 20:54:11,407][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen\_unicast\_24\_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]] [2015-02-05 20:54:11,408][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen\_unicast\_20\_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]] [2015-02-05 20:54:11,409][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen\_unicast\_27\_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]] [2015-02-05 20:54:11,413][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen\_unicast\_21\_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]] [2015-02-05 20:54:11,413][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen\_unicast\_26\_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]] [2015-02-05 20:54:11,413][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen\_unicast\_25\_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]] [2015-02-05 20:54:11,414][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen\_unicast\_23\_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]] [2015-02-05 20:54:11,414][TRACE][discovery.ec2] [d1b9ebf8c605] full ping responses: --\> ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]} --\> ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]} [2015-02-05 20:54:11,415][DEBUG][discovery.ec2] [d1b9ebf8c605] filtered ping responses: (filter\_client[true], filter\_data[false]) --\> ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]} --\> ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]} [2015-02-05 20:54:11,419][TRACE][discovery.ec2] [d1b9ebf8c605] joining master [d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true} [2015-02-05 20:54:11,434][DEBUG][discovery.zen.publish] [d1b9ebf8c605] received cluster state version 3 [2015-02-05 20:54:11,436][DEBUG][discovery.zen.fd] [d1b9ebf8c605] [master] restarting fault detection against master [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], reason [new cluster state received and we are monitoring the wrong master [null]] [2015-02-05 20:54:11,437][DEBUG][discovery.ec2] [d1b9ebf8c605] got first state from fresh master [9nNJp0wHSi2sa44Cyk37Dw] [2015-02-05 20:54:11,437][TRACE][discovery.ec2] [d1b9ebf8c605] updated cluster join cluster to [1] [2015-02-05 20:54:11,439][INFO][cluster.service] [d1b9ebf8c605] detected\_master [d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}, added {[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true},[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true},}, reason: zen-disco-receive(from master [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}]) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3a433856-2629-413f-a02f-0b9154302bb1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3a433856-2629-413f-a02f-0b9154302bb1%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
### Author: ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet) #### Post date: [February 6, 2015, 10:51am UTC](https://discuss.elastic.co/t/different-es-clusters-in-the-same-host/22006/2 "2015-02-06T10:51:30Z")
What we do this in AWS plugin: - get the IP address from AWS API (which obviously does know anything about the port used) address = instance.getPrivateIpAddress(); - transform this String to a TransportAddress which will basically be IP:9300: TransportAddress addresses = transportService.addressesFromString(address); - try to join IP:9300 which obviously answers but this is not the one you are expecting. I wonder if setting transport.tcp.port: 9302 could help here. By default this value is 9300-9399. -- David Pilato | Technical Advocate | [Elasticsearch.com](http://Elasticsearch.com) @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > Le 6 févr. 2015 à 01:23, Ricardo Alves dos Reis [ricardo.areis@gmail.com](mailto:ricardo.areis@gmail.com) a écrit : > > Hello folks, > > First of all, I'd like to explain my current necessity to bring this future trend out, some projects in my company have bean developed to use a lot of micro-services, unfortunately, is not uncommon that the mean of cpu usage in our cloud servers (AWS) are sustained in a lower degrees. > > To provisioning these resources easily all the components are executed in a docker containers and these stacks were represented as yaml file ([http://www.fig.sh](http://www.fig.sh)). In a group of servers I've more than one Elasticsearch clusters, containerized to use EC2 discovery and with a transport.publish\_port configured to avoid collision, however, only the first cluster in default transport port have a correct cluster formation. > > My guess is the transport.publish\_post is not used as pointer to discovery.zen.unicast to avoid unused transport port.. > > Thanks for any kind of help. > > #docker #aws #discovery > > fig.yml > > esemail: > image: XXX/elasticsearch:latest > volumes: > > - /dev/log:/dev/log > - /data0/email:/data/0 > - /data1/email:/data/1 > ports: > - "9202:9200" > - "9302:9300" > environment: > TRANSPORT\_PUBLISH\_PORT: 9302 > NETWORK\_PUBLISH\_HOST: _ec2_ > CLUSTER\_ROUTING\_ALLOCATION\_SAME\_SHARD\_HOST: "true" > ES\_ROOT\_LOG\_LEVEL: INFO > ES\_DISCOVERY\_LOG\_LEVEL: TRACE > ES\_LOG\_DIR: /var/log/elasticsearch > ES\_HEAP\_SIZE: 1g > ES\_DIRECT\_SIZE: 256m > ES\_JAVA\_OPTS: "-server -XX:+UseCompressedOops -XX:+DoEscapeAnalysis -XX:+AlwaysPreTouch -verbose:gc -Xloggc:/var/log/elasticsearch/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=512K" > MAX\_OPEN\_FILES: 65535 > MAX\_MAP\_COUNT: 262144 > CLUSTER\_NAME: email-es > NODE\_MASTER: "true" > NODE\_DATA: "true" > PATH\_DATA: /data/0/datasearch0,/data/1/datasearch0 > BOOTSTRAP\_MLOCKALL: "false" > DISCOVERY\_ZEN\_MCAST: "false" > DISCOVERY\_ZEN\_REJOIN\_ON\_MASTER\_GONE: "false" > DISCOVERY\_ZEN\_MINIMUM\_MASTER\_NODES: 2 > DISCOVERY\_TYPE: ec2 > DISCOVERY\_EC2\_ANY\_GROUP: "false" > DISCOVERY\_EC2\_GROUPS: db-prod-es-mail > DISCOVERY\_EC2\_AVAILABILITY\_ZONES: us-east-1a,us-east-1b,us-east-1d > DISCOVERY\_EC2\_HOST\_TYPE: private\_ip > DISCOVERY\_EC2\_PING\_TIMEOUT: 1m > GATEWAY\_RECOVER\_AFTER\_NODES: 2 > GATEWAY\_RECOVER\_AFTER\_TIME: 2m > GATEWAY\_EXPECTED\_NODES: 3 > AWS\_ELB\_NAME: ESEmailBionexo > AWS\_ACCESS\_KEY\_ID: XXXXX > AWS\_SECRET\_ACCESS\_KEY: XXXX > AWS\_DEFAULT\_REGION: us-east-1 > THREAD\_POOL\_SEARCH: 2000 > THREAD\_POOL\_INDEX: 2000 > THREAD\_POOL\_BULK: 2000 > > [ec2-user@ip-172-16-255-20 ~]$ sudo docker ps |grep elastic > d1b9ebf8c605 bionexo/elasticsearch:2015020502 "/bin/sh -c /bionexo 3 hours ago Up 3 hours 0.0.0.0:9202-\>9200/tcp, 0.0.0.0:9302-\>9300/tcp elasticsearch\_esemail\_1 > 088da31f943b bionexo/elasticsearch:2015020401 "/bin/sh -c /bionexo 35 hours ago Up 35 hours 0.0.0.0:9200-\>9200/tcp, 0.0.0.0:9300-\>9300/tcp elasticsearch\_eslogger\_1 > > [2015-02-05 20:53:09,834][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:09,853][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:09,854][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:10,637][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:10,637][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:23,833][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:25,837][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:26,618][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:35,724][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:38,034][TRACE][discovery.ec2] [d1b9ebf8c605] building dynamic unicast discovery nodes... > [2015-02-05 20:53:38,034][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-713a399d based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,034][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-7bd2f097 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,035][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-0b25abfa based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,035][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-83c39ca2 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: automation,GroupId: sg-90c510f5}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,035][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-7d49e897 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,035][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-fb9cc0d7 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,035][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-a72ef14d based on groups [{GroupName: corporativo,GroupId: sg-8f9c46ea}, {GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,035][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-54763907 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,035][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-b98e5e48 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,036][TRACE][discovery.ec2] [d1b9ebf8c605] adding i-b0384441, address 172.16.250.83, transport\_address inet[/172.16.250.83:9300] > [2015-02-05 20:53:38,036][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-84b324d7 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: demo\_apps,GroupId: sg-44c31d21}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,036][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-eb29db01 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,037][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-6cdd259d based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,037][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-53cabead based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,037][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-f5f21004 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,037][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-fd21a317 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,037][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-ebd815bb based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,037][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-81187b70 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,038][TRACE][discovery.ec2] [d1b9ebf8c605] adding i-4d52afb7, address 172.16.255.20, transport\_address inet[/172.16.255.20:9300] > [2015-02-05 20:53:38,039][TRACE][discovery.ec2] [d1b9ebf8c605] adding i-90f2367f, address 172.16.255.149, transport\_address inet[/172.16.255.149:9300] > [2015-02-05 20:53:38,039][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-b25d8e5f based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,039][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-4517fab4 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,039][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-573bf607 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: vpn,GroupId: sg-4fac3e2a}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,039][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-ca09f627 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: varnish,GroupId: sg-3815eb5c}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,040][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-a2728b53 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,040][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-fb222600 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,040][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-502433bc based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,040][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-2e871305 based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,040][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-3a7f91c0 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,040][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-19a469e3 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,040][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-e8038e04 based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail] > [2015-02-05 20:53:38,040][DEBUG][discovery.ec2] [d1b9ebf8c605] using dynamic discovery nodes [[#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]], [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]], [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]] > [2015-02-05 20:53:38,041][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] replacing [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]] with temp node [#zen\_unicast\_22\_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]] > [2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connecting (light) to [#zen\_unicast\_22\_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]] > [2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] replacing [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]] with temp node [#zen\_unicast\_23\_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]] > [2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connecting (light) to [#zen\_unicast\_23\_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]] > [2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] replacing [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]] with temp node [#zen\_unicast\_24\_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]] > [2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connected to [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]] > [2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [#zen\_unicast\_22\_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]], using \>=1.4.0 serialization > [2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connecting (light) to [#zen\_unicast\_24\_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]] > [2015-02-05 20:53:38,044][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connected to [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]] > [2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [#zen\_unicast\_23\_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]], using \>=1.4.0 serialization > [2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connected to [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]] > [2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [#zen\_unicast\_24\_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]], using \>=1.4.0 serialization > [2015-02-05 20:53:38,046][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true} > [2015-02-05 20:53:38,055][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [#zen\_unicast\_23\_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]: [ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[5], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true}], id[442], master [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], hasJoinedOnce [true], cluster\_name[logger]}] > [2015-02-05 20:53:38,054][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [#zen\_unicast\_22\_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]: [ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[5], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], id[454], master [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], hasJoinedOnce [true], cluster\_name[logger]}] > [2015-02-05 20:53:38,055][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] filtering out response from [088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}, not same cluster\_name [logger] > [2015-02-05 20:53:38,056][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] filtering out response from [1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true}, not same cluster\_name [logger] > [2015-02-05 20:53:38,057][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}: [ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[5], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[16], master [null], hasJoinedOnce [false], cluster\_name[email-es]}] > [2015-02-05 20:53:38,857][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:38,864][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:38,864][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:38,866][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:38,886][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:38,890][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:39,659][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:39,660][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:39,665][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:39,674][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:39,714][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:39,719][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:39,722][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:39,744][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:39,851][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:39,855][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:39,854][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:39,853][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:39,857][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:39,859][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:39,927][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:39,929][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:39,929][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:39,931][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:39,931][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:39,934][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:39,934][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:39,935][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:39,937][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:39,937][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:39,938][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,037][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,038][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,039][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,044][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,044][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,046][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,047][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,048][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,048][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,049][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,049][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,050][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,053][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,054][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,068][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,071][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,074][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,079][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,080][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,082][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,619][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,622][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:40,623][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,632][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:40,632][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:40,634][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,636][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:40,644][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:40,647][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:40,657][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,658][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:40,662][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,671][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,671][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,671][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,675][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:40,684][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:40,686][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:40,687][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:40,687][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:40,689][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:40,689][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:40,690][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:40,691][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:40,692][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:40,692][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:40,692][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:40,694][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:40,692][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:40,695][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:40,694][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:40,700][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:40,700][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:40,985][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:41,172][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:41,177][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:41,182][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:41,183][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:41,186][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:41,188][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:41,194][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:41,197][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:41,200][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:41,201][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:41,203][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:41,203][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:41,207][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:41,208][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:41,210][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:41,211][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:41,215][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:41,218][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:41,229][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:41,231][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:41,234][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:41,236][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:41,240][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:41,241][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:41,241][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:41,242][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:42,249][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:43,617][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:44,206][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:45,041][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [#zen\_unicast\_24\_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]: [ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[5], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[22b068e7698f][HZKmb28dQRO7Zc4QR-_qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true}], id[43], master [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], hasJoinedOnce [true], cluster\_name[logger]}] > [2015-02-05 20:53:45,041][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] filtering out response from [22b068e7698f][HZKmb28dQRO7Zc4QR-qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true}, not same cluster\_name [logger] > [2015-02-05 20:53:45,288][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:46,241][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:48,549][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:48,808][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:49,835][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:51,279][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:52,079][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:53,051][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:53,359][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:54,618][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:54,621][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:54,850][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:57,485][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:53:58,526][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:53:59,842][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:54:00,938][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:54:01,047][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:54:02,304][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:54:03,105][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:54:04,315][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:54:06,436][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:54:08,295][TRACE][discovery.ec2] [d1b9ebf8c605] building dynamic unicast discovery nodes... > [2015-02-05 20:54:08,297][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-713a399d based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,297][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-7bd2f097 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,297][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-0b25abfa based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,297][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-83c39ca2 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: automation,GroupId: sg-90c510f5}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,297][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-7d49e897 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,298][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-fb9cc0d7 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,298][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-a72ef14d based on groups [{GroupName: corporativo,GroupId: sg-8f9c46ea}, {GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,298][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-54763907 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,299][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-b98e5e48 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,301][TRACE][discovery.ec2] [d1b9ebf8c605] adding i-b0384441, address 172.16.250.83, transport\_address inet[/172.16.250.83:9300] > [2015-02-05 20:54:08,301][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-84b324d7 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: demo\_apps,GroupId: sg-44c31d21}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,301][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-eb29db01 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,301][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-6cdd259d based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,301][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-53cabead based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,302][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-f5f21004 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,302][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-fd21a317 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,302][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-ebd815bb based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,302][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-81187b70 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,304][TRACE][discovery.ec2] [d1b9ebf8c605] adding i-4d52afb7, address 172.16.255.20, transport\_address inet[/172.16.255.20:9300] > [2015-02-05 20:54:08,312][TRACE][discovery.ec2] [d1b9ebf8c605] adding i-90f2367f, address 172.16.255.149, transport\_address inet[/172.16.255.149:9300] > [2015-02-05 20:54:08,313][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-b25d8e5f based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,313][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-4517fab4 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,313][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-573bf607 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: vpn,GroupId: sg-4fac3e2a}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,315][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-ca09f627 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: varnish,GroupId: sg-3815eb5c}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,316][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-a2728b53 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,316][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-fb222600 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,316][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-502433bc based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,316][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-2e871305 based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,316][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-3a7f91c0 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,316][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-19a469e3 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,317][TRACE][discovery.ec2] [d1b9ebf8c605] filtering out instance i-e8038e04 based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail] > [2015-02-05 20:54:08,317][DEBUG][discovery.ec2] [d1b9ebf8c605] using dynamic discovery nodes [[#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]], [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]], [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]] > [2015-02-05 20:54:08,317][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] replacing [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]] with temp node [#zen\_unicast\_25#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]] > [2015-02-05 20:54:08,317][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connecting (light) to [#zen\_unicast\_25_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]] > [2015-02-05 20:54:08,318][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] replacing [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]] with temp node [#zen\_unicast\_26\_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]] > [2015-02-05 20:54:08,319][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connected to [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]] > [2015-02-05 20:54:08,319][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [#zen\_unicast\_25\_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]], using \>=1.4.0 serialization > [2015-02-05 20:54:08,322][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [#zen\_unicast\_25\_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]: [ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], id[457], master [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], hasJoinedOnce [true], cluster\_name[logger]}] > [2015-02-05 20:54:08,322][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] filtering out response from [088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}, not same cluster\_name [logger] > [2015-02-05 20:54:08,325][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connecting (light) to [#zen\_unicast\_26\_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]] > [2015-02-05 20:54:08,326][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] replacing [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]] with temp node [#zen\_unicast\_27\_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]] > [2015-02-05 20:54:08,327][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connecting (light) to [#zen\_unicast\_27\_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]] > [2015-02-05 20:54:08,327][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true} > [2015-02-05 20:54:08,328][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connected to [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]] > [2015-02-05 20:54:08,334][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [#zen\_unicast\_27\_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]], using \>=1.4.0 serialization > [2015-02-05 20:54:08,328][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connected to [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]] > [2015-02-05 20:54:08,334][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [#zen\_unicast\_26\_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]], using \>=1.4.0 serialization > [2015-02-05 20:54:08,341][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [#zen\_unicast\_26\_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]: [ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true}], id[445], master [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], hasJoinedOnce [true], cluster\_name[logger]}] > [2015-02-05 20:54:08,341][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] filtering out response from [1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true}, not same cluster\_name [logger] > [2015-02-05 20:54:08,342][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}: [ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[18], master [null], hasJoinedOnce [false], cluster\_name[email-es]}] > [2015-02-05 20:54:09,814][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:54:09,835][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:54:10,616][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:54:10,637][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:54:10,638][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m] > [2015-02-05 20:54:10,854][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:54:10,857][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry > [2015-02-05 20:54:11,404][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [#zen\_unicast\_27\_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]: [ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]}, ping\_response{node [[22b068e7698f][HZKmb28dQRO7Zc4QR-_qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true}], id[46], master [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], hasJoinedOnce [true], cluster\_name[logger]}] > [2015-02-05 20:54:11,405][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] filtering out response from [22b068e7698f][HZKmb28dQRO7Zc4QR-qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true}, not same cluster\_name [logger] > [2015-02-05 20:54:11,405][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen\_unicast\_19#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]] > [2015-02-05 20:54:11,406][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen\_unicast\_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]] > [2015-02-05 20:54:11,407][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen\_unicast\_24\_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]] > [2015-02-05 20:54:11,408][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen\_unicast\_20\_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]] > [2015-02-05 20:54:11,409][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen\_unicast\_27\_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]] > [2015-02-05 20:54:11,413][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen\_unicast\_21\_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]] > [2015-02-05 20:54:11,413][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen\_unicast\_26\_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]] > [2015-02-05 20:54:11,413][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen\_unicast\_25\_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]] > [2015-02-05 20:54:11,414][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen\_unicast\_23\_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]] > [2015-02-05 20:54:11,414][TRACE][discovery.ec2] [d1b9ebf8c605] full ping responses: > --\> ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]} > --\> ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]} > [2015-02-05 20:54:11,415][DEBUG][discovery.ec2] [d1b9ebf8c605] filtered ping responses: (filter\_client[true], filter\_data[false]) > --\> ping\_response{node [[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]} > --\> ping\_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster\_name[email-es]} > [2015-02-05 20:54:11,419][TRACE][discovery.ec2] [d1b9ebf8c605] joining master [d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true} > [2015-02-05 20:54:11,434][DEBUG][discovery.zen.publish] [d1b9ebf8c605] received cluster state version 3 > [2015-02-05 20:54:11,436][DEBUG][discovery.zen.fd] [d1b9ebf8c605] [master] restarting fault detection against master [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], reason [new cluster state received and we are monitoring the wrong master [null]] > [2015-02-05 20:54:11,437][DEBUG][discovery.ec2] [d1b9ebf8c605] got first state from fresh master [9nNJp0wHSi2sa44Cyk37Dw] > [2015-02-05 20:54:11,437][TRACE][discovery.ec2] [d1b9ebf8c605] updated cluster join cluster to [1] > [2015-02-05 20:54:11,439][INFO][cluster.service] [d1b9ebf8c605] detected\_master [d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}, added {[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true},[bc7cbb4635f9][c\_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true},}, reason: zen-disco-receive(from master [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}]) > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3a433856-2629-413f-a02f-0b9154302bb1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3a433856-2629-413f-a02f-0b9154302bb1%40googlegroups.com) [https://groups.google.com/d/msgid/elasticsearch/3a433856-2629-413f-a02f-0b9154302bb1%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/3a433856-2629-413f-a02f-0b9154302bb1%40googlegroups.com?utm_medium=email&utm_source=footer). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/091AE197-22B3-495A-832B-757941F6A8D4%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/091AE197-22B3-495A-832B-757941F6A8D4%40pilato.fr). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
### Author: ![Ricardo\_Reis](https://avatars.discourse-cdn.com/v4/letter/r/54ee81/32.png) [@Ricardo\_Reis](https://discuss.elastic.co/u/Ricardo_Reis) #### Post date: [February 12, 2015, 1:33pm UTC](https://discuss.elastic.co/t/different-es-clusters-in-the-same-host/22006/3 "2015-02-12T13:33:10Z")
Hello David, It's really solve my problem 😉 Thanks Ricardo Reis Em sexta-feira, 6 de fevereiro de 2015 08:52:03 UTC-2, David Pilato escreveu: > What we do this in AWS plugin: > > - get the IP address from AWS API (which obviously does know anything > about the port used) > address = instance.getPrivateIpAddress(); > > - transform this String to a TransportAddress which will basically be > IP:9300: > TransportAddress addresses > = transportService.addressesFromString(address); > > - try to join IP:9300 which obviously answers but this is not the one you > are expecting. > > I wonder if setting transport.tcp.port: 9302 could help here. > By default this value is 9300-9399. > > -- > _David Pilato_ | _Technical Advocate_ | _[Elasticsearch.com](http://Elasticsearch.com) > [http://Elasticsearch.com](http://Elasticsearch.com)_ > @dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr > [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) | @scrutmydocs > [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs) > > Le 6 févr. 2015 à 01:23, Ricardo Alves dos Reis \<[ricard...@gmail.com](mailto:ricard...@gmail.com) > \\> a écrit : > > Hello folks, > > First of all, I'd like to explain my current necessity to bring this > future trend out, some projects in my company have bean developed to use a > lot of micro-services, unfortunately, is not uncommon that the mean of cpu > usage in our cloud servers (AWS) are sustained in a lower degrees. > > To provisioning these resources easily all the components are executed in > a docker containers and these stacks were represented as yaml file ( > [http://www.fig.sh](http://www.fig.sh)). In a group of servers I've more than one > Elasticsearch clusters, containerized to use EC2 discovery and with a > transport.publish\_port configured to avoid collision, however, only the > first cluster in default transport port have a correct cluster formation. > > My guess is the transport.publish\_post is not used as pointer to > discovery.zen.unicast to avoid unused transport port.. > > Thanks for any kind of help. > > #docker #aws #discovery > > fig.yml > > esemail: > > image: XXX/elasticsearch:latest > > volumes: > > - /dev/log:/dev/log > > - /data0/email:/data/0 > > - /data1/email:/data/1 > > ports: > > - "9202:9200" > > - "9302:9300" > > environment: > > TRANSPORT\_PUBLISH\_PORT: 9302 > > NETWORK\_PUBLISH\_HOST: _ec2_ > > CLUSTER\_ROUTING\_ALLOCATION\_SAME\_SHARD\_HOST: "true" > > ES\_ROOT\_LOG\_LEVEL: INFO > > ES\_DISCOVERY\_LOG\_LEVEL: TRACE > > ES\_LOG\_DIR: /var/log/elasticsearch > > ES\_HEAP\_SIZE: 1g > > ES\_DIRECT\_SIZE: 256m > > ES\_JAVA\_OPTS: "-server -XX:+UseCompressedOops -XX:+DoEscapeAnalysis > -XX:+AlwaysPreTouch -verbose:gc -Xloggc:/var/log/elasticsearch/gc.log > -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=512K" > > MAX\_OPEN\_FILES: 65535 > > MAX\_MAP\_COUNT: 262144 > > CLUSTER\_NAME: email-es > > NODE\_MASTER: "true" > > NODE\_DATA: "true" > > PATH\_DATA: /data/0/datasearch0,/data/1/datasearch0 > > BOOTSTRAP\_MLOCKALL: "false" > > DISCOVERY\_ZEN\_MCAST: "false" > > DISCOVERY\_ZEN\_REJOIN\_ON\_MASTER\_GONE: "false" > > DISCOVERY\_ZEN\_MINIMUM\_MASTER\_NODES: 2 > > DISCOVERY\_TYPE: ec2 > > DISCOVERY\_EC2\_ANY\_GROUP: "false" > > DISCOVERY\_EC2\_GROUPS: db-prod-es-mail > > DISCOVERY\_EC2\_AVAILABILITY\_ZONES: us-east-1a,us-east-1b,us-east-1d > > DISCOVERY\_EC2\_HOST\_TYPE: private\_ip > > DISCOVERY\_EC2\_PING\_TIMEOUT: 1m > > GATEWAY\_RECOVER\_AFTER\_NODES: 2 > > GATEWAY\_RECOVER\_AFTER\_TIME: 2m > > GATEWAY\_EXPECTED\_NODES: 3 > > AWS\_ELB\_NAME: ESEmailBionexo > > AWS\_ACCESS\_KEY\_ID: XXXXX > > AWS\_SECRET\_ACCESS\_KEY: XXXX > > AWS\_DEFAULT\_REGION: us-east-1 > > THREAD\_POOL\_SEARCH: 2000 > > THREAD\_POOL\_INDEX: 2000 > > THREAD\_POOL\_BULK: 2000 > > [ec2-user@ip-172-16-255-20 ~]$ sudo docker ps |grep elastic > > d1b9ebf8c605 bionexo/elasticsearch:2015020502 "/bin/sh -c /bionexo 3 hours > ago Up 3 hours 0.0.0.0:9202-\>9200/tcp, 0.0.0.0:9302-\>9300/tcp > elasticsearch\_esemail\_1 > > 088da31f943b bionexo/elasticsearch:2015020401 "/bin/sh -c /bionexo 35 > hours ago Up 35 hours 0.0.0.0:9200-\>9200/tcp, 0.0.0.0:9300-\>9300/tcp > elasticsearch\_eslogger\_1 > > [2015-02-05 20:53:09,834][DEBUG][action.admin.indices.create] > [d1b9ebf8c605] no known master node, scheduling a retry > > [2015-02-05 20:53:09,853][DEBUG][action.admin.indices.create] > [d1b9ebf8c605] observer: timeout notification from cluster service. timeout > setting [1m], time since start [1m] > > [2015-02-05 20:53:09,854][DEBUG][action.admin.indices.create] > [d1b9ebf8c605] observer: timeout notification from cluster service. timeout > setting [1m], time since start [1m] > > [2015-02-05 20:53:10,637][DEBUG][action.admin.indices.create] > [d1b9ebf8c605] no known master node, scheduling a retry > > [2015-02-05 20:53:10,637][DEBUG][action.admin.indices.create] > [d1b9ebf8c605] no known master node, scheduling a retry > > [2015-02-05 20:53:23,833][DEBUG][action.admin.indices.create] > [d1b9ebf8c605] no known master node, scheduling a retry > > [2015-02-05 20:53:25,837][DEBUG][action.admin.indices.create] > [d1b9ebf8c605] observer: timeout notification from cluster service. timeout > setting [1m], time since start [1m] > > [2015-02-05 20:53:26,618][DEBUG][action.admin.indices.create] > [d1b9ebf8c605] no known master node, scheduling a retry > > [2015-02-05 20:53:35,724][DEBUG][action.admin.indices.create] > [d1b9ebf8c605] observer: timeout notification from cluster service. timeout > setting [1m], time since start [1m] > > [2015-02-05 20:53:38,034][TRACE][discovery.ec2] [d1b9ebf8c605] building > dynamic unicast discovery nodes... > > [2015-02-05 20:53:38,034][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-713a399d based on groups [{GroupName: default,GroupId: > sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: > host-healthcheck,GroupId: sg-73df7e17}], does not include all of > [db-prod-es-mail] > > [2015-02-05 20:53:38,034][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-7bd2f097 based on groups [{GroupName: default,GroupId: > sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: > host-healthcheck,GroupId: sg-73df7e17}], does not include all of > [db-prod-es-mail] > > [2015-02-05 20:53:38,035][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-0b25abfa based on groups [{GroupName: default,GroupId: > sg-62e42407}], does not include all of [db-prod-es-mail] > > [2015-02-05 20:53:38,035][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-83c39ca2 based on groups [{GroupName: default,GroupId: > sg-62e42407}, {GroupName: automation,GroupId: sg-90c510f5}], does not > include all of [db-prod-es-mail] > > [2015-02-05 20:53:38,035][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-7d49e897 based on groups [{GroupName: default,GroupId: > sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not > include all of [db-prod-es-mail] > > [2015-02-05 20:53:38,035][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-fb9cc0d7 based on groups [{GroupName: default,GroupId: > sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}], does not > include all of [db-prod-es-mail] > > [2015-02-05 20:53:38,035][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-a72ef14d based on groups [{GroupName: corporativo,GroupId: > sg-8f9c46ea}, {GroupName: default,GroupId: sg-62e42407}], does not include > all of [db-prod-es-mail] > > [2015-02-05 20:53:38,035][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-54763907 based on groups [{GroupName: default,GroupId: > sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}], does not > include all of [db-prod-es-mail] > > [2015-02-05 20:53:38,035][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-b98e5e48 based on groups [{GroupName: default,GroupId: > sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: > host-healthcheck,GroupId: sg-73df7e17}], does not include all of > [db-prod-es-mail] > > [2015-02-05 20:53:38,036][TRACE][discovery.ec2] [d1b9ebf8c605] adding > i-b0384441, address 172.16.250.83, transport\_address inet[/ > 172.16.250.83:9300] > > [2015-02-05 20:53:38,036][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-84b324d7 based on groups [{GroupName: default,GroupId: > sg-62e42407}, {GroupName: demo\_apps,GroupId: sg-44c31d21}], does not > include all of [db-prod-es-mail] > > [2015-02-05 20:53:38,036][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-eb29db01 based on groups [{GroupName: default,GroupId: > sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}], does not > include all of [db-prod-es-mail] > > [2015-02-05 20:53:38,037][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-6cdd259d based on groups [{GroupName: default,GroupId: > sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: > host-healthcheck,GroupId: sg-73df7e17}], does not include all of > [db-prod-es-mail] > > [2015-02-05 20:53:38,037][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-53cabead based on groups [{GroupName: default,GroupId: > sg-62e42407}], does not include all of [db-prod-es-mail] > > [2015-02-05 20:53:38,037][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-f5f21004 based on groups [{GroupName: default,GroupId: > sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: > host-healthcheck,GroupId: sg-73df7e17}], does not include all of > [db-prod-es-mail] > > [2015-02-05 20:53:38,037][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-fd21a317 based on groups [{GroupName: default,GroupId: > sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}], does not > include all of [db-prod-es-mail] > > [2015-02-05 20:53:38,037][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-ebd815bb based on groups [{GroupName: default,GroupId: > sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not > include all of [db-prod-es-mail] > > [2015-02-05 20:53:38,037][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-81187b70 based on groups [{GroupName: default,GroupId: > sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: > host-healthcheck,GroupId: sg-73df7e17}], does not include all of > [db-prod-es-mail] > > [2015-02-05 20:53:38,038][TRACE][discovery.ec2] [d1b9ebf8c605] adding > i-4d52afb7, address 172.16.255.20, transport\_address inet[/ > 172.16.255.20:9300] > > [2015-02-05 20:53:38,039][TRACE][discovery.ec2] [d1b9ebf8c605] adding > i-90f2367f, address 172.16.255.149, transport\_address inet[/ > 172.16.255.149:9300] > > [2015-02-05 20:53:38,039][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-b25d8e5f based on groups [{GroupName: default,GroupId: > sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not > include all of [db-prod-es-mail] > > [2015-02-05 20:53:38,039][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-4517fab4 based on groups [{GroupName: default,GroupId: > sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: > host-healthcheck,GroupId: sg-73df7e17}], does not include all of > [db-prod-es-mail] > > [2015-02-05 20:53:38,039][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-573bf607 based on groups [{GroupName: default,GroupId: > sg-62e42407}, {GroupName: vpn,GroupId: sg-4fac3e2a}], does not include all > of [db-prod-es-mail] > > [2015-02-05 20:53:38,039][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-ca09f627 based on groups [{GroupName: default,GroupId: > sg-62e42407}, {GroupName: varnish,GroupId: sg-3815eb5c}], does not include > all of [db-prod-es-mail] > > [2015-02-05 20:53:38,040][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-a2728b53 based on groups [{GroupName: default,GroupId: > sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: > host-healthcheck,GroupId: sg-73df7e17}], does not include all of > [db-prod-es-mail] > > [2015-02-05 20:53:38,040][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-fb222600 based on groups [{GroupName: default,GroupId: > sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: > host-healthcheck,GroupId: sg-73df7e17}], does not include all of > [db-prod-es-mail] > > [2015-02-05 20:53:38,040][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-502433bc based on groups [{GroupName: default,GroupId: > sg-62e42407}, {GroupName: prod\_apps,GroupId: sg-620d6b07}, {GroupName: > host-healthcheck,GroupId: sg-73df7e17}], does not include all of > [db-prod-es-mail] > > [2015-02-05 20:53:38,040][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-2e871305 based on groups [{GroupName: default,GroupId: > sg-62e42407}], does not include all of [db-prod-es-mail] > > [2015-02-05 20:53:38,040][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-3a7f91c0 based on groups [{GroupName: default,GroupId: > sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not > include all of [db-prod-es-mail] > > [2015-02-05 20:53:38,040][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-19a469e3 based on groups [{GroupName: default,GroupId: > sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not > include all of [db-prod-es-mail] > > [2015-02-05 20:53:38,040][TRACE][discovery.ec2] [d1b9ebf8c605] filtering > out instance i-e8038e04 based on groups [{GroupName: default,GroupId: > sg-62e42407}], does not include all of [db-prod-es-mail] > > [2015-02-05 20:53:38,040][DEBUG][discovery.ec2] [d1b9ebf8c605] using > dynamic discovery nodes > [[#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]], > [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]], > [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]] > > [2015-02-05 20:53:38,041][TRACE][discovery.zen.ping.unicast] > [d1b9ebf8c605] replacing [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/ > 172.16.255.20:9300]] with temp node > [#zen\_unicast\_22\_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/ > 172.16.255.20:9300]] > > [2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast] > [d1b9ebf8c605] [3] connecting (light) to > [#zen\_unicast\_22\_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/ > 172.16.255.20:9300]] > > [2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast] > [d1b9ebf8c605] replacing [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/ > 172.16.255.149:9300]] with temp node > [#zen\_unicast\_23\_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/ > 172.16.255.149:9300]] > > [2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast] > [d1b9ebf8c605] [3] connecting (light) to > [#zen\_unicast\_23\_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/ > 172.16.255.149:9300]] > > [2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] > [d1b9ebf8c605] replacing [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/ > 172.16.250.83:9300]] with temp node > [#zen\_unicast\_24\_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/ > 172.16.250.83:9300]] > > [2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] > [d1b9ebf8c605] [3] connected to [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/ > 172.16.255.20:9300]] > > [2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] > [d1b9ebf8c605] [3] sending to > [#zen\_unicast\_22\_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]], > using \>=1.4.0 serialization > > [2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] > [d1b9ebf8c605] [3] connecting (light) to > [#zen\_unicast\_24\_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/ > 172.16.250.83:9300]] > > [2015-02-05 20:53:38,044][TRACE][discovery.zen.ping.unicast] > [d1b9ebf8c605] [3] connected to [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/ > 172.16.255.149:9300]] > > [2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast] > [d1b9ebf8c605] [3] sending to > [#zen\_unicast\_23\_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]], > using \>=1.4.0 serialization > > [2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast] > [d1b9ebf8c605] [3] connected to [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/ > 172.16.250.83:9300]] > > [2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast] > [d1b9ebf8c605] [3] sending to > [#zen\_unicast\_24\_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]], > using \>=1.4.0 serialization > > [2015-02-05 20:53:38,046][TRACE][discovery.zen.ping.unicast] > [d1b9ebf8c605] [3] sending to > [d1b9ebf8c605][KT-HWB\_xRpWhis\_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true} > > [2015-02-05 20:53:38,055][TRACE][discovery.zen.ping.unicast] > [d1b9ebf8c605] > > ... -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/92322f0d-c420-48f0-baf7-fa69934ef1d1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/92322f0d-c420-48f0-baf7-fa69934ef1d1%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system) #### Post date: [July 6, 2017, 12:33am UTC](https://discuss.elastic.co/t/different-es-clusters-in-the-same-host/22006/4 "2017-07-06T00:33:08Z")