Different ES clusters in the same host

Hello folks,

First of all, I'd like to explain my current necessity to bring this
future trend out, some projects in my company have bean developed to use a
lot of micro-services, unfortunately, is not uncommon that the mean of cpu
usage in our cloud servers (AWS) are sustained in a lower degrees.

To provisioning these resources easily all the components are executed in
a docker containers and these stacks were represented as yaml file
(http://www.fig.sh). In a group of servers I've more than one Elasticsearch
clusters, containerized to use EC2 discovery and with a
transport.publish_port configured to avoid collision, however, only the
first cluster in default transport port have a correct cluster formation.

My guess is the transport.publish_post is not used as pointer to
discovery.zen.unicast to avoid unused transport port..

Thanks for any kind of help.

#docker #aws #discovery

fig.yml

esemail:

image: XXX/elasticsearch:latest

volumes:

  • /dev/log:/dev/log

  • /data0/email:/data/0

  • /data1/email:/data/1

ports:

  • "9202:9200"

  • "9302:9300"

environment:

TRANSPORT_PUBLISH_PORT: 9302

NETWORK_PUBLISH_HOST: ec2

CLUSTER_ROUTING_ALLOCATION_SAME_SHARD_HOST: "true"

ES_ROOT_LOG_LEVEL: INFO

ES_DISCOVERY_LOG_LEVEL: TRACE

ES_LOG_DIR: /var/log/elasticsearch

ES_HEAP_SIZE: 1g

ES_DIRECT_SIZE: 256m

ES_JAVA_OPTS: "-server -XX:+UseCompressedOops -XX:+DoEscapeAnalysis
-XX:+AlwaysPreTouch -verbose:gc -Xloggc:/var/log/elasticsearch/gc.log
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=512K"

MAX_OPEN_FILES: 65535

MAX_MAP_COUNT: 262144

CLUSTER_NAME: email-es

NODE_MASTER: "true"

NODE_DATA: "true"

PATH_DATA: /data/0/datasearch0,/data/1/datasearch0

BOOTSTRAP_MLOCKALL: "false"

DISCOVERY_ZEN_MCAST: "false"

DISCOVERY_ZEN_REJOIN_ON_MASTER_GONE: "false"

DISCOVERY_ZEN_MINIMUM_MASTER_NODES: 2

DISCOVERY_TYPE: ec2

DISCOVERY_EC2_ANY_GROUP: "false"

DISCOVERY_EC2_GROUPS: db-prod-es-mail

DISCOVERY_EC2_AVAILABILITY_ZONES: us-east-1a,us-east-1b,us-east-1d

DISCOVERY_EC2_HOST_TYPE: private_ip

DISCOVERY_EC2_PING_TIMEOUT: 1m

GATEWAY_RECOVER_AFTER_NODES: 2

GATEWAY_RECOVER_AFTER_TIME: 2m

GATEWAY_EXPECTED_NODES: 3

AWS_ELB_NAME: ESEmailBionexo

AWS_ACCESS_KEY_ID: XXXXX

AWS_SECRET_ACCESS_KEY: XXXX

AWS_DEFAULT_REGION: us-east-1

THREAD_POOL_SEARCH: 2000

THREAD_POOL_INDEX: 2000

THREAD_POOL_BULK: 2000

[ec2-user@ip-172-16-255-20 ~]$ sudo docker ps |grep elastic

d1b9ebf8c605 bionexo/elasticsearch:2015020502 "/bin/sh -c /bionexo 3 hours
ago Up 3 hours 0.0.0.0:9202->9200/tcp, 0.0.0.0:9302->9300/tcp
elasticsearch_esemail_1

088da31f943b bionexo/elasticsearch:2015020401 "/bin/sh -c /bionexo 35 hours
ago Up 35 hours 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp
elasticsearch_eslogger_1

[2015-02-05 20:53:09,834][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:09,853][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:09,854][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:10,637][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:10,637][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:23,833][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:25,837][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:26,618][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:35,724][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:38,034][TRACE][discovery.ec2 ] [d1b9ebf8c605] building
dynamic unicast discovery nodes...

[2015-02-05 20:53:38,034][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-713a399d based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,034][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-7bd2f097 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-0b25abfa based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-83c39ca2 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: automation,GroupId: sg-90c510f5}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-7d49e897 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-fb9cc0d7 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-a72ef14d based on groups [{GroupName: corporativo,GroupId:
sg-8f9c46ea}, {GroupName: default,GroupId: sg-62e42407}], does not include
all of [db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-54763907 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-b98e5e48 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,036][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding
i-b0384441, address 172.16.250.83, transport_address
inet[/172.16.250.83:9300]

[2015-02-05 20:53:38,036][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-84b324d7 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: demo_apps,GroupId: sg-44c31d21}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,036][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-eb29db01 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-6cdd259d based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-53cabead based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]

[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-f5f21004 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-fd21a317 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-ebd815bb based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-81187b70 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,038][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding
i-4d52afb7, address 172.16.255.20, transport_address
inet[/172.16.255.20:9300]

[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding
i-90f2367f, address 172.16.255.149, transport_address
inet[/172.16.255.149:9300]

[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-b25d8e5f based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-4517fab4 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-573bf607 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: vpn,GroupId: sg-4fac3e2a}], does not include all
of [db-prod-es-mail]

[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-ca09f627 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: varnish,GroupId: sg-3815eb5c}], does not include
all of [db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-a2728b53 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-fb222600 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-502433bc based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-2e871305 based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-3a7f91c0 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-19a469e3 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-e8038e04 based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]

[2015-02-05 20:53:38,040][DEBUG][discovery.ec2 ] [d1b9ebf8c605] using
dynamic discovery nodes
[[#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]],
[#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]],
[#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]]

[2015-02-05 20:53:38,041][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
replacing [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]]
with temp node
[#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]

[2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connecting (light) to
[#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]

[2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
replacing [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]
with temp node
[#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]

[2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connecting (light) to
[#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]

[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
replacing [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]]
with temp node
[#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]

[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connected to
[#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]]

[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] sending to
[#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]],
using >=1.4.0 serialization

[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connecting (light) to
[#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]

[2015-02-05 20:53:38,044][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connected to
[#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]

[2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] sending to
[#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]],
using >=1.4.0 serialization

[2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connected to
[#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]]

[2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] sending to
[#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]],
using >=1.4.0 serialization

[2015-02-05 20:53:38,046][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] sending to
[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}

[2015-02-05 20:53:38,055][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] received response from
[#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]:
[ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[5], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true}],
id[442], master
[[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}],
hasJoinedOnce [true], cluster_name[logger]}]

[2015-02-05 20:53:38,054][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] received response from
[#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]:
[ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[5], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}],
id[454], master
[[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}],
hasJoinedOnce [true], cluster_name[logger]}]

[2015-02-05 20:53:38,055][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] filtering out response from
[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true},
not same cluster_name [logger]

[2015-02-05 20:53:38,056][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] filtering out response from
[1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true},
not same cluster_name [logger]

[2015-02-05 20:53:38,057][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] received response from
[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}:
[ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[5], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[16], master [null], hasJoinedOnce [false], cluster_name[email-es]}]

[2015-02-05 20:53:38,857][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:38,864][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:38,864][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:38,866][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:38,886][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:38,890][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,659][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,660][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,665][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,674][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,714][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,719][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,722][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,744][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,851][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:39,855][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,854][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:39,853][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:39,857][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:39,859][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,927][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,929][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,929][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,931][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,931][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,934][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,934][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,935][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,937][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,937][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,938][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,037][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,038][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,039][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,044][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,044][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,046][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,047][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,048][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,048][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,049][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,049][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,050][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,053][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,054][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,068][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,071][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,074][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,079][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,080][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,082][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,619][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,622][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,623][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,632][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,632][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,634][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,636][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,644][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,647][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,657][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,658][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,662][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,671][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,671][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,671][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,675][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,684][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,686][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,687][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,687][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,689][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,689][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,690][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,691][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,692][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,692][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,692][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,694][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,692][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,695][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,694][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,700][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,700][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,985][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:41,172][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,177][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,182][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,183][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,186][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,188][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,194][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,197][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,200][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,201][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,203][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,203][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,207][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,208][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,210][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,211][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,215][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,218][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,229][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,231][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,234][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,236][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,240][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,241][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,241][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,242][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:42,249][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:43,617][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:44,206][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:45,041][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] received response from
[#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]:
[ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[5], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[22b068e7698f][HZKmb28dQRO7Zc4QR-_qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true}],
id[43], master
[[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}],
hasJoinedOnce [true], cluster_name[logger]}]

[2015-02-05 20:53:45,041][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] filtering out response from
[22b068e7698f][HZKmb28dQRO7Zc4QR-_qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true},
not same cluster_name [logger]

[2015-02-05 20:53:45,288][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:46,241][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:48,549][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:48,808][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:49,835][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:51,279][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:52,079][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:53,051][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:53,359][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:54,618][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:54,621][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:54,850][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:57,485][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:58,526][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:59,842][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:54:00,938][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:54:01,047][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:54:02,304][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:54:03,105][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:54:04,315][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:54:06,436][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:54:08,295][TRACE][discovery.ec2 ] [d1b9ebf8c605] building
dynamic unicast discovery nodes...

[2015-02-05 20:54:08,297][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-713a399d based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:54:08,297][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-7bd2f097 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:54:08,297][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-0b25abfa based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]

[2015-02-05 20:54:08,297][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-83c39ca2 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: automation,GroupId: sg-90c510f5}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:54:08,297][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-7d49e897 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:54:08,298][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-fb9cc0d7 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:54:08,298][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-a72ef14d based on groups [{GroupName: corporativo,GroupId:
sg-8f9c46ea}, {GroupName: default,GroupId: sg-62e42407}], does not include
all of [db-prod-es-mail]

[2015-02-05 20:54:08,298][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-54763907 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:54:08,299][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-b98e5e48 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:54:08,301][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding
i-b0384441, address 172.16.250.83, transport_address
inet[/172.16.250.83:9300]

[2015-02-05 20:54:08,301][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-84b324d7 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: demo_apps,GroupId: sg-44c31d21}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:54:08,301][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-eb29db01 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:54:08,301][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-6cdd259d based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:54:08,301][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-53cabead based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]

[2015-02-05 20:54:08,302][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-f5f21004 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:54:08,302][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-fd21a317 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:54:08,302][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-ebd815bb based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:54:08,302][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-81187b70 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:54:08,304][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding
i-4d52afb7, address 172.16.255.20, transport_address
inet[/172.16.255.20:9300]

[2015-02-05 20:54:08,312][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding
i-90f2367f, address 172.16.255.149, transport_address
inet[/172.16.255.149:9300]

[2015-02-05 20:54:08,313][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-b25d8e5f based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:54:08,313][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-4517fab4 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:54:08,313][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-573bf607 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: vpn,GroupId: sg-4fac3e2a}], does not include all
of [db-prod-es-mail]

[2015-02-05 20:54:08,315][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-ca09f627 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: varnish,GroupId: sg-3815eb5c}], does not include
all of [db-prod-es-mail]

[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-a2728b53 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-fb222600 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-502433bc based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-2e871305 based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]

[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-3a7f91c0 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-19a469e3 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:54:08,317][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-e8038e04 based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]

[2015-02-05 20:54:08,317][DEBUG][discovery.ec2 ] [d1b9ebf8c605] using
dynamic discovery nodes
[[#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]],
[#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]],
[#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]]

[2015-02-05 20:54:08,317][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
replacing [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]]
with temp node
[#zen_unicast_25_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]

[2015-02-05 20:54:08,317][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connecting (light) to
[#zen_unicast_25_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]

[2015-02-05 20:54:08,318][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
replacing [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]
with temp node
[#zen_unicast_26_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]

[2015-02-05 20:54:08,319][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connected to
[#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]]

[2015-02-05 20:54:08,319][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] sending to
[#zen_unicast_25_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]],
using >=1.4.0 serialization

[2015-02-05 20:54:08,322][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] received response from
[#zen_unicast_25_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]:
[ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}],
id[457], master
[[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}],
hasJoinedOnce [true], cluster_name[logger]}]

[2015-02-05 20:54:08,322][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] filtering out response from
[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true},
not same cluster_name [logger]

[2015-02-05 20:54:08,325][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connecting (light) to
[#zen_unicast_26_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]

[2015-02-05 20:54:08,326][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
replacing [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]]
with temp node
[#zen_unicast_27_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]

[2015-02-05 20:54:08,327][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connecting (light) to
[#zen_unicast_27_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]

[2015-02-05 20:54:08,327][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] sending to
[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}

[2015-02-05 20:54:08,328][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connected to
[#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]]

[2015-02-05 20:54:08,334][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] sending to
[#zen_unicast_27_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]],
using >=1.4.0 serialization

[2015-02-05 20:54:08,328][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connected to
[#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]

[2015-02-05 20:54:08,334][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] sending to
[#zen_unicast_26_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]],
using >=1.4.0 serialization

[2015-02-05 20:54:08,341][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] received response from
[#zen_unicast_26_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]:
[ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true}],
id[445], master
[[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}],
hasJoinedOnce [true], cluster_name[logger]}]

[2015-02-05 20:54:08,341][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] filtering out response from
[1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true},
not same cluster_name [logger]

[2015-02-05 20:54:08,342][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] received response from
[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}:
[ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[18], master [null], hasJoinedOnce [false], cluster_name[email-es]}]

[2015-02-05 20:54:09,814][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:54:09,835][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:54:10,616][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:54:10,637][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:54:10,638][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:54:10,854][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:54:10,857][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:54:11,404][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] received response from
[#zen_unicast_27_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]:
[ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[22b068e7698f][HZKmb28dQRO7Zc4QR-_qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true}],
id[46], master
[[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}],
hasJoinedOnce [true], cluster_name[logger]}]

[2015-02-05 20:54:11,405][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] filtering out response from
[22b068e7698f][HZKmb28dQRO7Zc4QR-_qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true},
not same cluster_name [logger]

[2015-02-05 20:54:11,405][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_19_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]

[2015-02-05 20:54:11,406][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]

[2015-02-05 20:54:11,407][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]

[2015-02-05 20:54:11,408][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_20_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]

[2015-02-05 20:54:11,409][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_27_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]

[2015-02-05 20:54:11,413][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_21_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]

[2015-02-05 20:54:11,413][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_26_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]

[2015-02-05 20:54:11,413][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_25_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]

[2015-02-05 20:54:11,414][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]

[2015-02-05 20:54:11,414][TRACE][discovery.ec2 ] [d1b9ebf8c605] full ping
responses:

--> ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}

--> ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}

[2015-02-05 20:54:11,415][DEBUG][discovery.ec2 ] [d1b9ebf8c605] filtered
ping responses: (filter_client[true], filter_data[false])

--> ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}

--> ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}

[2015-02-05 20:54:11,419][TRACE][discovery.ec2 ] [d1b9ebf8c605] joining
master
[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}

[2015-02-05 20:54:11,434][DEBUG][discovery.zen.publish ] [d1b9ebf8c605]
received cluster state version 3

[2015-02-05 20:54:11,436][DEBUG][discovery.zen.fd ] [d1b9ebf8c605] [master]
restarting fault detection against master
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
reason [new cluster state received and we are monitoring the wrong master
[null]]

[2015-02-05 20:54:11,437][DEBUG][discovery.ec2 ] [d1b9ebf8c605] got first
state from fresh master [9nNJp0wHSi2sa44Cyk37Dw]

[2015-02-05 20:54:11,437][TRACE][discovery.ec2 ] [d1b9ebf8c605] updated
cluster join cluster to [1]

[2015-02-05 20:54:11,439][INFO ][cluster.service ] [d1b9ebf8c605]
detected_master
[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true},
added
{[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true},[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true},},
reason: zen-disco-receive(from master
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}])

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3a433856-2629-413f-a02f-0b9154302bb1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

What we do this in AWS plugin:

  • get the IP address from AWS API (which obviously does know anything about the port used)
    address = instance.getPrivateIpAddress();

  • transform this String to a TransportAddress which will basically be IP:9300:
    TransportAddress[] addresses = transportService.addressesFromString(address);

  • try to join IP:9300 which obviously answers but this is not the one you are expecting.

I wonder if setting transport.tcp.port: 9302 could help here.
By default this value is 9300-9399.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr https://twitter.com/elasticsearchfr | @scrutmydocs https://twitter.com/scrutmydocs

Le 6 févr. 2015 à 01:23, Ricardo Alves dos Reis ricardo.areis@gmail.com a écrit :

Hello folks,

First of all, I'd like to explain my current necessity to bring this future trend out, some projects in my company have bean developed to use a lot of micro-services, unfortunately, is not uncommon that the mean of cpu usage in our cloud servers (AWS) are sustained in a lower degrees.

To provisioning these resources easily all the components are executed in a docker containers and these stacks were represented as yaml file (http://www.fig.sh). In a group of servers I've more than one Elasticsearch clusters, containerized to use EC2 discovery and with a transport.publish_port configured to avoid collision, however, only the first cluster in default transport port have a correct cluster formation.

My guess is the transport.publish_post is not used as pointer to discovery.zen.unicast to avoid unused transport port..

Thanks for any kind of help.

#docker #aws #discovery

fig.yml

esemail:
image: XXX/elasticsearch:latest
volumes:

  • /dev/log:/dev/log
  • /data0/email:/data/0
  • /data1/email:/data/1
    ports:
  • "9202:9200"
  • "9302:9300"
    environment:
    TRANSPORT_PUBLISH_PORT: 9302
    NETWORK_PUBLISH_HOST: ec2
    CLUSTER_ROUTING_ALLOCATION_SAME_SHARD_HOST: "true"
    ES_ROOT_LOG_LEVEL: INFO
    ES_DISCOVERY_LOG_LEVEL: TRACE
    ES_LOG_DIR: /var/log/elasticsearch
    ES_HEAP_SIZE: 1g
    ES_DIRECT_SIZE: 256m
    ES_JAVA_OPTS: "-server -XX:+UseCompressedOops -XX:+DoEscapeAnalysis -XX:+AlwaysPreTouch -verbose:gc -Xloggc:/var/log/elasticsearch/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=512K"
    MAX_OPEN_FILES: 65535
    MAX_MAP_COUNT: 262144
    CLUSTER_NAME: email-es
    NODE_MASTER: "true"
    NODE_DATA: "true"
    PATH_DATA: /data/0/datasearch0,/data/1/datasearch0
    BOOTSTRAP_MLOCKALL: "false"
    DISCOVERY_ZEN_MCAST: "false"
    DISCOVERY_ZEN_REJOIN_ON_MASTER_GONE: "false"
    DISCOVERY_ZEN_MINIMUM_MASTER_NODES: 2
    DISCOVERY_TYPE: ec2
    DISCOVERY_EC2_ANY_GROUP: "false"
    DISCOVERY_EC2_GROUPS: db-prod-es-mail
    DISCOVERY_EC2_AVAILABILITY_ZONES: us-east-1a,us-east-1b,us-east-1d
    DISCOVERY_EC2_HOST_TYPE: private_ip
    DISCOVERY_EC2_PING_TIMEOUT: 1m
    GATEWAY_RECOVER_AFTER_NODES: 2
    GATEWAY_RECOVER_AFTER_TIME: 2m
    GATEWAY_EXPECTED_NODES: 3
    AWS_ELB_NAME: ESEmailBionexo
    AWS_ACCESS_KEY_ID: XXXXX
    AWS_SECRET_ACCESS_KEY: XXXX
    AWS_DEFAULT_REGION: us-east-1
    THREAD_POOL_SEARCH: 2000
    THREAD_POOL_INDEX: 2000
    THREAD_POOL_BULK: 2000

[ec2-user@ip-172-16-255-20 ~]$ sudo docker ps |grep elastic
d1b9ebf8c605 bionexo/elasticsearch:2015020502 "/bin/sh -c /bionexo 3 hours ago Up 3 hours 0.0.0.0:9202->9200/tcp, 0.0.0.0:9302->9300/tcp elasticsearch_esemail_1
088da31f943b bionexo/elasticsearch:2015020401 "/bin/sh -c /bionexo 35 hours ago Up 35 hours 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp elasticsearch_eslogger_1

[2015-02-05 20:53:09,834][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:09,853][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:09,854][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:10,637][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:10,637][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:23,833][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:25,837][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:26,618][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:35,724][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:38,034][TRACE][discovery.ec2 ] [d1b9ebf8c605] building dynamic unicast discovery nodes...
[2015-02-05 20:53:38,034][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-713a399d based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,034][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-7bd2f097 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-0b25abfa based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-83c39ca2 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: automation,GroupId: sg-90c510f5}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-7d49e897 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-fb9cc0d7 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-a72ef14d based on groups [{GroupName: corporativo,GroupId: sg-8f9c46ea}, {GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-54763907 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-b98e5e48 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,036][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding i-b0384441, address 172.16.250.83, transport_address inet[/172.16.250.83:9300]
[2015-02-05 20:53:38,036][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-84b324d7 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: demo_apps,GroupId: sg-44c31d21}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,036][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-eb29db01 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-6cdd259d based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-53cabead based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-f5f21004 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-fd21a317 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-ebd815bb based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-81187b70 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,038][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding i-4d52afb7, address 172.16.255.20, transport_address inet[/172.16.255.20:9300]
[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding i-90f2367f, address 172.16.255.149, transport_address inet[/172.16.255.149:9300]
[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-b25d8e5f based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-4517fab4 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-573bf607 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: vpn,GroupId: sg-4fac3e2a}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-ca09f627 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: varnish,GroupId: sg-3815eb5c}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-a2728b53 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-fb222600 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-502433bc based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-2e871305 based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-3a7f91c0 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-19a469e3 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-e8038e04 based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,040][DEBUG][discovery.ec2 ] [d1b9ebf8c605] using dynamic discovery nodes [[#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]], [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]], [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]]
[2015-02-05 20:53:38,041][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] replacing [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]] with temp node [#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connecting (light) to [#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] replacing [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]] with temp node [#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connecting (light) to [#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] replacing [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]] with temp node [#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connected to [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]], using >=1.4.0 serialization
[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connecting (light) to [#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:53:38,044][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connected to [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]], using >=1.4.0 serialization
[2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connected to [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]], using >=1.4.0 serialization
[2015-02-05 20:53:38,046][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}
[2015-02-05 20:53:38,055][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]: [ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[5], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true}], id[442], master [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], hasJoinedOnce [true], cluster_name[logger]}]
[2015-02-05 20:53:38,054][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]: [ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[5], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], id[454], master [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], hasJoinedOnce [true], cluster_name[logger]}]
[2015-02-05 20:53:38,055][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] filtering out response from [088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}, not same cluster_name [logger]
[2015-02-05 20:53:38,056][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] filtering out response from [1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true}, not same cluster_name [logger]
[2015-02-05 20:53:38,057][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}: [ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[5], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[16], master [null], hasJoinedOnce [false], cluster_name[email-es]}]
[2015-02-05 20:53:38,857][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:38,864][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:38,864][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:38,866][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:38,886][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:38,890][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,659][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,660][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,665][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,674][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,714][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,719][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,722][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,744][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,851][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:39,855][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,854][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:39,853][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:39,857][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:39,859][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,927][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,929][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,929][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,931][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,931][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,934][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,934][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,935][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,937][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,937][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,938][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,037][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,038][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,039][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,044][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,044][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,046][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,047][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,048][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,048][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,049][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,049][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,050][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,053][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,054][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,068][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,071][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,074][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,079][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,080][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,082][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,619][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,622][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,623][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,632][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,632][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,634][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,636][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,644][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,647][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,657][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,658][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,662][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,671][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,671][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,671][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,675][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,684][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,686][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,687][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,687][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,689][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,689][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,690][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,691][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,692][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,692][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,692][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,694][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,692][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,695][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,694][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,700][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,700][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,985][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:41,172][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,177][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,182][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,183][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,186][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,188][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,194][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,197][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,200][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,201][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,203][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,203][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,207][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,208][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,210][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,211][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,215][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,218][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,229][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,231][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,234][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,236][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,240][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,241][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,241][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,242][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:42,249][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:43,617][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:44,206][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:45,041][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]: [ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[5], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[22b068e7698f][HZKmb28dQRO7Zc4QR-qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true}], id[43], master [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], hasJoinedOnce [true], cluster_name[logger]}]
[2015-02-05 20:53:45,041][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] filtering out response from [22b068e7698f][HZKmb28dQRO7Zc4QR-qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true}, not same cluster_name [logger]
[2015-02-05 20:53:45,288][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:46,241][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:48,549][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:48,808][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:49,835][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:51,279][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:52,079][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:53,051][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:53,359][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:54,618][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:54,621][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:54,850][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:57,485][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:58,526][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:59,842][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:54:00,938][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:54:01,047][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:54:02,304][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:54:03,105][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:54:04,315][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:54:06,436][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:54:08,295][TRACE][discovery.ec2 ] [d1b9ebf8c605] building dynamic unicast discovery nodes...
[2015-02-05 20:54:08,297][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-713a399d based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,297][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-7bd2f097 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,297][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-0b25abfa based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,297][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-83c39ca2 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: automation,GroupId: sg-90c510f5}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,297][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-7d49e897 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,298][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-fb9cc0d7 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,298][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-a72ef14d based on groups [{GroupName: corporativo,GroupId: sg-8f9c46ea}, {GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,298][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-54763907 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,299][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-b98e5e48 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,301][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding i-b0384441, address 172.16.250.83, transport_address inet[/172.16.250.83:9300]
[2015-02-05 20:54:08,301][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-84b324d7 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: demo_apps,GroupId: sg-44c31d21}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,301][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-eb29db01 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,301][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-6cdd259d based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,301][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-53cabead based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,302][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-f5f21004 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,302][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-fd21a317 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,302][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-ebd815bb based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,302][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-81187b70 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,304][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding i-4d52afb7, address 172.16.255.20, transport_address inet[/172.16.255.20:9300]
[2015-02-05 20:54:08,312][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding i-90f2367f, address 172.16.255.149, transport_address inet[/172.16.255.149:9300]
[2015-02-05 20:54:08,313][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-b25d8e5f based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,313][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-4517fab4 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,313][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-573bf607 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: vpn,GroupId: sg-4fac3e2a}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,315][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-ca09f627 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: varnish,GroupId: sg-3815eb5c}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-a2728b53 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-fb222600 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-502433bc based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-2e871305 based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-3a7f91c0 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-19a469e3 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,317][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-e8038e04 based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,317][DEBUG][discovery.ec2 ] [d1b9ebf8c605] using dynamic discovery nodes [[#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]], [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]], [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]]
[2015-02-05 20:54:08,317][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] replacing [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]] with temp node [#zen_unicast_25
#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:54:08,317][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connecting (light) to [#zen_unicast_25
#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:54:08,318][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] replacing [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]] with temp node [#zen_unicast_26_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:54:08,319][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connected to [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:54:08,319][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [#zen_unicast_25_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]], using >=1.4.0 serialization
[2015-02-05 20:54:08,322][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [#zen_unicast_25_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]: [ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], id[457], master [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], hasJoinedOnce [true], cluster_name[logger]}]
[2015-02-05 20:54:08,322][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] filtering out response from [088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}, not same cluster_name [logger]
[2015-02-05 20:54:08,325][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connecting (light) to [#zen_unicast_26_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:54:08,326][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] replacing [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]] with temp node [#zen_unicast_27_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:54:08,327][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connecting (light) to [#zen_unicast_27_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:54:08,327][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}
[2015-02-05 20:54:08,328][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connected to [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:54:08,334][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [#zen_unicast_27_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]], using >=1.4.0 serialization
[2015-02-05 20:54:08,328][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connected to [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:54:08,334][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [#zen_unicast_26_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]], using >=1.4.0 serialization
[2015-02-05 20:54:08,341][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [#zen_unicast_26_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]: [ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true}], id[445], master [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], hasJoinedOnce [true], cluster_name[logger]}]
[2015-02-05 20:54:08,341][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] filtering out response from [1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true}, not same cluster_name [logger]
[2015-02-05 20:54:08,342][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}: [ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[18], master [null], hasJoinedOnce [false], cluster_name[email-es]}]
[2015-02-05 20:54:09,814][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:54:09,835][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:54:10,616][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:54:10,637][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:54:10,638][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:54:10,854][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:54:10,857][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:54:11,404][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [#zen_unicast_27_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]: [ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[22b068e7698f][HZKmb28dQRO7Zc4QR-qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true}], id[46], master [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], hasJoinedOnce [true], cluster_name[logger]}]
[2015-02-05 20:54:11,405][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] filtering out response from [22b068e7698f][HZKmb28dQRO7Zc4QR-qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true}, not same cluster_name [logger]
[2015-02-05 20:54:11,405][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen_unicast_19
#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:54:11,406][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen_unicast_22
#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:54:11,407][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:54:11,408][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen_unicast_20_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:54:11,409][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen_unicast_27_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:54:11,413][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen_unicast_21_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:54:11,413][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen_unicast_26_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:54:11,413][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen_unicast_25_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:54:11,414][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:54:11,414][TRACE][discovery.ec2 ] [d1b9ebf8c605] full ping responses:
--> ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}
--> ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}
[2015-02-05 20:54:11,415][DEBUG][discovery.ec2 ] [d1b9ebf8c605] filtered ping responses: (filter_client[true], filter_data[false])
--> ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}
--> ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}
[2015-02-05 20:54:11,419][TRACE][discovery.ec2 ] [d1b9ebf8c605] joining master [d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}
[2015-02-05 20:54:11,434][DEBUG][discovery.zen.publish ] [d1b9ebf8c605] received cluster state version 3
[2015-02-05 20:54:11,436][DEBUG][discovery.zen.fd ] [d1b9ebf8c605] [master] restarting fault detection against master [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], reason [new cluster state received and we are monitoring the wrong master [null]]
[2015-02-05 20:54:11,437][DEBUG][discovery.ec2 ] [d1b9ebf8c605] got first state from fresh master [9nNJp0wHSi2sa44Cyk37Dw]
[2015-02-05 20:54:11,437][TRACE][discovery.ec2 ] [d1b9ebf8c605] updated cluster join cluster to [1]
[2015-02-05 20:54:11,439][INFO ][cluster.service ] [d1b9ebf8c605] detected_master [d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}, added {[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true},[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true},}, reason: zen-disco-receive(from master [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}])

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3a433856-2629-413f-a02f-0b9154302bb1%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/3a433856-2629-413f-a02f-0b9154302bb1%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/091AE197-22B3-495A-832B-757941F6A8D4%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Hello David,

It's really solve my problem :wink:

Thanks

Ricardo Reis

Em sexta-feira, 6 de fevereiro de 2015 08:52:03 UTC-2, David Pilato
escreveu:

What we do this in AWS plugin:

  • get the IP address from AWS API (which obviously does know anything
    about the port used)
    address = instance.getPrivateIpAddress();

  • transform this String to a TransportAddress which will basically be
    IP:9300:
    TransportAddress[] addresses
    = transportService.addressesFromString(address);

  • try to join IP:9300 which obviously answers but this is not the one you
    are expecting.

I wonder if setting transport.tcp.port: 9302 could help here.
By default this value is 9300-9399.

--
David Pilato | Technical Advocate | Elasticsearch.com
http://Elasticsearch.com

@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 6 févr. 2015 à 01:23, Ricardo Alves dos Reis <ricard...@gmail.com
<javascript:>> a écrit :

Hello folks,

First of all, I'd like to explain my current necessity to bring this
future trend out, some projects in my company have bean developed to use a
lot of micro-services, unfortunately, is not uncommon that the mean of cpu
usage in our cloud servers (AWS) are sustained in a lower degrees.

To provisioning these resources easily all the components are executed in
a docker containers and these stacks were represented as yaml file (
http://www.fig.sh). In a group of servers I've more than one
Elasticsearch clusters, containerized to use EC2 discovery and with a
transport.publish_port configured to avoid collision, however, only the
first cluster in default transport port have a correct cluster formation.

My guess is the transport.publish_post is not used as pointer to
discovery.zen.unicast to avoid unused transport port..

Thanks for any kind of help.

#docker #aws #discovery

fig.yml

esemail:

image: XXX/elasticsearch:latest

volumes:

  • /dev/log:/dev/log

  • /data0/email:/data/0

  • /data1/email:/data/1

ports:

  • "9202:9200"

  • "9302:9300"

environment:

TRANSPORT_PUBLISH_PORT: 9302

NETWORK_PUBLISH_HOST: ec2

CLUSTER_ROUTING_ALLOCATION_SAME_SHARD_HOST: "true"

ES_ROOT_LOG_LEVEL: INFO

ES_DISCOVERY_LOG_LEVEL: TRACE

ES_LOG_DIR: /var/log/elasticsearch

ES_HEAP_SIZE: 1g

ES_DIRECT_SIZE: 256m

ES_JAVA_OPTS: "-server -XX:+UseCompressedOops -XX:+DoEscapeAnalysis
-XX:+AlwaysPreTouch -verbose:gc -Xloggc:/var/log/elasticsearch/gc.log
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=512K"

MAX_OPEN_FILES: 65535

MAX_MAP_COUNT: 262144

CLUSTER_NAME: email-es

NODE_MASTER: "true"

NODE_DATA: "true"

PATH_DATA: /data/0/datasearch0,/data/1/datasearch0

BOOTSTRAP_MLOCKALL: "false"

DISCOVERY_ZEN_MCAST: "false"

DISCOVERY_ZEN_REJOIN_ON_MASTER_GONE: "false"

DISCOVERY_ZEN_MINIMUM_MASTER_NODES: 2

DISCOVERY_TYPE: ec2

DISCOVERY_EC2_ANY_GROUP: "false"

DISCOVERY_EC2_GROUPS: db-prod-es-mail

DISCOVERY_EC2_AVAILABILITY_ZONES: us-east-1a,us-east-1b,us-east-1d

DISCOVERY_EC2_HOST_TYPE: private_ip

DISCOVERY_EC2_PING_TIMEOUT: 1m

GATEWAY_RECOVER_AFTER_NODES: 2

GATEWAY_RECOVER_AFTER_TIME: 2m

GATEWAY_EXPECTED_NODES: 3

AWS_ELB_NAME: ESEmailBionexo

AWS_ACCESS_KEY_ID: XXXXX

AWS_SECRET_ACCESS_KEY: XXXX

AWS_DEFAULT_REGION: us-east-1

THREAD_POOL_SEARCH: 2000

THREAD_POOL_INDEX: 2000

THREAD_POOL_BULK: 2000

[ec2-user@ip-172-16-255-20 ~]$ sudo docker ps |grep elastic

d1b9ebf8c605 bionexo/elasticsearch:2015020502 "/bin/sh -c /bionexo 3 hours
ago Up 3 hours 0.0.0.0:9202->9200/tcp, 0.0.0.0:9302->9300/tcp
elasticsearch_esemail_1

088da31f943b bionexo/elasticsearch:2015020401 "/bin/sh -c /bionexo 35
hours ago Up 35 hours 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp
elasticsearch_eslogger_1

[2015-02-05 20:53:09,834][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:09,853][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:09,854][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:10,637][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:10,637][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:23,833][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:25,837][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:26,618][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:35,724][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:38,034][TRACE][discovery.ec2 ] [d1b9ebf8c605] building
dynamic unicast discovery nodes...

[2015-02-05 20:53:38,034][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-713a399d based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,034][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-7bd2f097 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-0b25abfa based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-83c39ca2 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: automation,GroupId: sg-90c510f5}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-7d49e897 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-fb9cc0d7 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-a72ef14d based on groups [{GroupName: corporativo,GroupId:
sg-8f9c46ea}, {GroupName: default,GroupId: sg-62e42407}], does not include
all of [db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-54763907 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-b98e5e48 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,036][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding
i-b0384441, address 172.16.250.83, transport_address inet[/
172.16.250.83:9300]

[2015-02-05 20:53:38,036][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-84b324d7 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: demo_apps,GroupId: sg-44c31d21}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,036][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-eb29db01 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-6cdd259d based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-53cabead based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]

[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-f5f21004 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-fd21a317 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-ebd815bb based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-81187b70 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,038][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding
i-4d52afb7, address 172.16.255.20, transport_address inet[/
172.16.255.20:9300]

[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding
i-90f2367f, address 172.16.255.149, transport_address inet[/
172.16.255.149:9300]

[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-b25d8e5f based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-4517fab4 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-573bf607 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: vpn,GroupId: sg-4fac3e2a}], does not include all
of [db-prod-es-mail]

[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-ca09f627 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: varnish,GroupId: sg-3815eb5c}], does not include
all of [db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-a2728b53 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-fb222600 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-502433bc based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-2e871305 based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-3a7f91c0 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-19a469e3 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-e8038e04 based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]

[2015-02-05 20:53:38,040][DEBUG][discovery.ec2 ] [d1b9ebf8c605] using
dynamic discovery nodes
[[#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]],
[#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]],
[#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]]

[2015-02-05 20:53:38,041][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605] replacing [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/
172.16.255.20:9300]] with temp node
[#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/
172.16.255.20:9300]]

[2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605] [3] connecting (light) to
[#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/
172.16.255.20:9300]]

[2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605] replacing [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/
172.16.255.149:9300]] with temp node
[#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/
172.16.255.149:9300]]

[2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605] [3] connecting (light) to
[#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/
172.16.255.149:9300]]

[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605] replacing [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/
172.16.250.83:9300]] with temp node
[#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/
172.16.250.83:9300]]

[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605] [3] connected to [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/
172.16.255.20:9300]]

[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605] [3] sending to
[#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]],
using >=1.4.0 serialization

[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605] [3] connecting (light) to
[#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/
172.16.250.83:9300]]

[2015-02-05 20:53:38,044][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605] [3] connected to [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/
172.16.255.149:9300]]

[2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605] [3] sending to
[#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]],
using >=1.4.0 serialization

[2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605] [3] connected to [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/
172.16.250.83:9300]]

[2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605] [3] sending to
[#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]],
using >=1.4.0 serialization

[2015-02-05 20:53:38,046][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605] [3] sending to
[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}

[2015-02-05 20:53:38,055][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605]

...

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/92322f0d-c420-48f0-baf7-fa69934ef1d1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.