Different ES clusters in the same host

Ricardo_Alves_dos_Re · February 6, 2015, 12:23am

Hello folks,

First of all, I'd like to explain my current necessity to bring this
future trend out, some projects in my company have bean developed to use a
lot of micro-services, unfortunately, is not uncommon that the mean of cpu
usage in our cloud servers (AWS) are sustained in a lower degrees.

To provisioning these resources easily all the components are executed in
a docker containers and these stacks were represented as yaml file
(http://www.fig.sh). In a group of servers I've more than one Elasticsearch
clusters, containerized to use EC2 discovery and with a
transport.publish_port configured to avoid collision, however, only the
first cluster in default transport port have a correct cluster formation.

My guess is the transport.publish_post is not used as pointer to
discovery.zen.unicast to avoid unused transport port..

Thanks for any kind of help.

#docker #aws #discovery

fig.yml

esemail:

image: XXX/elasticsearch:latest

volumes:

/dev/log:/dev/log
/data0/email:/data/0
/data1/email:/data/1

ports:

"9202:9200"
"9302:9300"

environment:

TRANSPORT_PUBLISH_PORT: 9302

NETWORK_PUBLISH_HOST: ec2

CLUSTER_ROUTING_ALLOCATION_SAME_SHARD_HOST: "true"

ES_ROOT_LOG_LEVEL: INFO

ES_DISCOVERY_LOG_LEVEL: TRACE

ES_LOG_DIR: /var/log/elasticsearch

ES_HEAP_SIZE: 1g

ES_DIRECT_SIZE: 256m

ES_JAVA_OPTS: "-server -XX:+UseCompressedOops -XX:+DoEscapeAnalysis
-XX:+AlwaysPreTouch -verbose:gc -Xloggc:/var/log/elasticsearch/gc.log
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=512K"

MAX_OPEN_FILES: 65535

MAX_MAP_COUNT: 262144

CLUSTER_NAME: email-es

NODE_MASTER: "true"

NODE_DATA: "true"

PATH_DATA: /data/0/datasearch0,/data/1/datasearch0

BOOTSTRAP_MLOCKALL: "false"

DISCOVERY_ZEN_MCAST: "false"

DISCOVERY_ZEN_REJOIN_ON_MASTER_GONE: "false"

DISCOVERY_ZEN_MINIMUM_MASTER_NODES: 2

DISCOVERY_TYPE: ec2

DISCOVERY_EC2_ANY_GROUP: "false"

DISCOVERY_EC2_GROUPS: db-prod-es-mail

DISCOVERY_EC2_AVAILABILITY_ZONES: us-east-1a,us-east-1b,us-east-1d

DISCOVERY_EC2_HOST_TYPE: private_ip

DISCOVERY_EC2_PING_TIMEOUT: 1m

GATEWAY_RECOVER_AFTER_NODES: 2

GATEWAY_RECOVER_AFTER_TIME: 2m

GATEWAY_EXPECTED_NODES: 3

AWS_ELB_NAME: ESEmailBionexo

AWS_ACCESS_KEY_ID: XXXXX

AWS_SECRET_ACCESS_KEY: XXXX

AWS_DEFAULT_REGION: us-east-1

THREAD_POOL_SEARCH: 2000

THREAD_POOL_INDEX: 2000

THREAD_POOL_BULK: 2000

[ec2-user@ip-172-16-255-20 ~]$ sudo docker ps |grep elastic

d1b9ebf8c605 bionexo/elasticsearch:2015020502 "/bin/sh -c /bionexo 3 hours
ago Up 3 hours 0.0.0.0:9202->9200/tcp, 0.0.0.0:9302->9300/tcp
elasticsearch_esemail_1

088da31f943b bionexo/elasticsearch:2015020401 "/bin/sh -c /bionexo 35 hours
ago Up 35 hours 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp
elasticsearch_eslogger_1

[2015-02-05 20:53:09,834][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:09,853][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:09,854][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:10,637][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:23,833][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:25,837][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:26,618][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:35,724][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:38,034][TRACE][discovery.ec2 ] [d1b9ebf8c605] building
dynamic unicast discovery nodes...

[2015-02-05 20:53:38,034][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-713a399d based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,034][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-7bd2f097 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-0b25abfa based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-83c39ca2 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: automation,GroupId: sg-90c510f5}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-7d49e897 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-fb9cc0d7 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-a72ef14d based on groups [{GroupName: corporativo,GroupId:
sg-8f9c46ea}, {GroupName: default,GroupId: sg-62e42407}], does not include
all of [db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-54763907 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-b98e5e48 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,036][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding
i-b0384441, address 172.16.250.83, transport_address
inet[/172.16.250.83:9300]

[2015-02-05 20:53:38,036][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-84b324d7 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: demo_apps,GroupId: sg-44c31d21}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,036][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-eb29db01 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-6cdd259d based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-53cabead based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]

[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-f5f21004 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-fd21a317 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-ebd815bb based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-81187b70 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,038][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding
i-4d52afb7, address 172.16.255.20, transport_address
inet[/172.16.255.20:9300]

[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding
i-90f2367f, address 172.16.255.149, transport_address
inet[/172.16.255.149:9300]

[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-b25d8e5f based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-4517fab4 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-573bf607 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: vpn,GroupId: sg-4fac3e2a}], does not include all
of [db-prod-es-mail]

[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-ca09f627 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: varnish,GroupId: sg-3815eb5c}], does not include
all of [db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-a2728b53 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-fb222600 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-502433bc based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-2e871305 based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-3a7f91c0 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-19a469e3 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-e8038e04 based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]

[2015-02-05 20:53:38,040][DEBUG][discovery.ec2 ] [d1b9ebf8c605] using
dynamic discovery nodes
[[#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]],
[#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]],
[#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]]

[2015-02-05 20:53:38,041][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
replacing [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]]
with temp node
[#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]

[2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connecting (light) to
[#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]

[2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
replacing [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]
with temp node
[#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]

[2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connecting (light) to
[#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]

[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
replacing [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]]
with temp node
[#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]

[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connected to
[#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]]

[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] sending to
[#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]],
using >=1.4.0 serialization

[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connecting (light) to
[#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]

[2015-02-05 20:53:38,044][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connected to
[#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]

[2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] sending to
[#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]],
using >=1.4.0 serialization

[2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connected to
[#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]]

[2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] sending to
[#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]],
using >=1.4.0 serialization

[2015-02-05 20:53:38,046][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] sending to
[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}

[2015-02-05 20:53:38,055][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] received response from
[#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]:
[ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[5], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true}],
id[442], master
[[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}],
hasJoinedOnce [true], cluster_name[logger]}]

[2015-02-05 20:53:38,054][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] received response from
[#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]:
[ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[5], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}],
id[454], master
[[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}],
hasJoinedOnce [true], cluster_name[logger]}]

[2015-02-05 20:53:38,055][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] filtering out response from
[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true},
not same cluster_name [logger]

[2015-02-05 20:53:38,056][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] filtering out response from
[1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true},
not same cluster_name [logger]

[2015-02-05 20:53:38,057][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] received response from
[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}:
[ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[5], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[16], master [null], hasJoinedOnce [false], cluster_name[email-es]}]

[2015-02-05 20:53:38,857][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:38,864][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:38,866][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:38,886][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:38,890][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,659][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,660][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,665][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,674][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,714][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,719][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,722][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,744][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,851][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:39,855][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,854][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:39,853][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:39,857][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:39,859][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,927][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,929][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,931][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,934][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,935][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,937][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:39,938][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,037][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,038][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,039][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,044][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,046][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,047][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,048][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,049][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,050][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,053][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,054][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,068][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,071][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,074][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,079][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,080][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,082][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,619][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,622][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,623][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,632][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,634][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,636][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,644][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,647][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,657][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,658][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,662][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,671][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,675][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:40,684][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,686][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,687][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,689][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,690][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,691][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,692][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,694][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,692][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,695][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,694][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,700][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:40,985][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:41,172][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,177][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,182][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,183][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,186][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,188][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,194][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,197][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,200][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,201][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,203][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,207][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,208][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,210][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,211][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,215][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,218][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,229][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,231][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,234][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,236][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,240][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,241][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:41,242][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:42,249][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:43,617][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:44,206][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:45,041][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] received response from
[#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]:
[ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[5], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[22b068e7698f][HZKmb28dQRO7Zc4QR-_qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true}],
id[43], master
[[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}],
hasJoinedOnce [true], cluster_name[logger]}]

[2015-02-05 20:53:45,041][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] filtering out response from
[22b068e7698f][HZKmb28dQRO7Zc4QR-_qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true},
not same cluster_name [logger]

[2015-02-05 20:53:45,288][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:46,241][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:48,549][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:48,808][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:49,835][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:51,279][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:52,079][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:53,051][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:53,359][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:54,618][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:54,621][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:54,850][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:57,485][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:58,526][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:59,842][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:54:00,938][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:54:01,047][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:54:02,304][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:54:03,105][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:54:04,315][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:54:06,436][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:54:08,295][TRACE][discovery.ec2 ] [d1b9ebf8c605] building
dynamic unicast discovery nodes...

[2015-02-05 20:54:08,297][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-713a399d based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:54:08,297][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-7bd2f097 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:54:08,297][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-0b25abfa based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]

[2015-02-05 20:54:08,297][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-83c39ca2 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: automation,GroupId: sg-90c510f5}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:54:08,297][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-7d49e897 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:54:08,298][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-fb9cc0d7 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:54:08,298][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-a72ef14d based on groups [{GroupName: corporativo,GroupId:
sg-8f9c46ea}, {GroupName: default,GroupId: sg-62e42407}], does not include
all of [db-prod-es-mail]

[2015-02-05 20:54:08,298][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-54763907 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:54:08,299][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-b98e5e48 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:54:08,301][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding
i-b0384441, address 172.16.250.83, transport_address
inet[/172.16.250.83:9300]

[2015-02-05 20:54:08,301][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-84b324d7 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: demo_apps,GroupId: sg-44c31d21}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:54:08,301][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-eb29db01 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:54:08,301][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-6cdd259d based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:54:08,301][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-53cabead based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]

[2015-02-05 20:54:08,302][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-f5f21004 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:54:08,302][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-fd21a317 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:54:08,302][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-ebd815bb based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:54:08,302][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-81187b70 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:54:08,304][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding
i-4d52afb7, address 172.16.255.20, transport_address
inet[/172.16.255.20:9300]

[2015-02-05 20:54:08,312][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding
i-90f2367f, address 172.16.255.149, transport_address
inet[/172.16.255.149:9300]

[2015-02-05 20:54:08,313][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-b25d8e5f based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:54:08,313][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-4517fab4 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:54:08,313][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-573bf607 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: vpn,GroupId: sg-4fac3e2a}], does not include all
of [db-prod-es-mail]

[2015-02-05 20:54:08,315][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-ca09f627 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: varnish,GroupId: sg-3815eb5c}], does not include
all of [db-prod-es-mail]

[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-a2728b53 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-fb222600 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-502433bc based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-2e871305 based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]

[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-3a7f91c0 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-19a469e3 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:54:08,317][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-e8038e04 based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]

[2015-02-05 20:54:08,317][DEBUG][discovery.ec2 ] [d1b9ebf8c605] using
dynamic discovery nodes
[[#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]],
[#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]],
[#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]]

[2015-02-05 20:54:08,317][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
replacing [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]]
with temp node
[#zen_unicast_25_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]

[2015-02-05 20:54:08,317][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connecting (light) to
[#zen_unicast_25_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]

[2015-02-05 20:54:08,318][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
replacing [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]
with temp node
[#zen_unicast_26_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]

[2015-02-05 20:54:08,319][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connected to
[#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]]

[2015-02-05 20:54:08,319][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] sending to
[#zen_unicast_25_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]],
using >=1.4.0 serialization

[2015-02-05 20:54:08,322][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] received response from
[#zen_unicast_25_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]:
[ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}],
id[457], master
[[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}],
hasJoinedOnce [true], cluster_name[logger]}]

[2015-02-05 20:54:08,322][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] filtering out response from
[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true},
not same cluster_name [logger]

[2015-02-05 20:54:08,325][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connecting (light) to
[#zen_unicast_26_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]

[2015-02-05 20:54:08,326][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
replacing [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]]
with temp node
[#zen_unicast_27_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]

[2015-02-05 20:54:08,327][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connecting (light) to
[#zen_unicast_27_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]

[2015-02-05 20:54:08,327][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] sending to
[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}

[2015-02-05 20:54:08,328][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connected to
[#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]]

[2015-02-05 20:54:08,334][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] sending to
[#zen_unicast_27_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]],
using >=1.4.0 serialization

[2015-02-05 20:54:08,328][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connected to
[#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]

[2015-02-05 20:54:08,334][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] sending to
[#zen_unicast_26_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]],
using >=1.4.0 serialization

[2015-02-05 20:54:08,341][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] received response from
[#zen_unicast_26_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]:
[ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true}],
id[445], master
[[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}],
hasJoinedOnce [true], cluster_name[logger]}]

[2015-02-05 20:54:08,341][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] filtering out response from
[1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true},
not same cluster_name [logger]

[2015-02-05 20:54:08,342][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] received response from
[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}:
[ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[18], master [null], hasJoinedOnce [false], cluster_name[email-es]}]

[2015-02-05 20:54:09,814][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:54:09,835][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:54:10,616][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:54:10,637][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:54:10,638][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:54:10,854][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:54:10,857][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:54:11,404][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] received response from
[#zen_unicast_27_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]:
[ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[22b068e7698f][HZKmb28dQRO7Zc4QR-_qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true}],
id[46], master
[[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}],
hasJoinedOnce [true], cluster_name[logger]}]

[2015-02-05 20:54:11,405][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] filtering out response from
[22b068e7698f][HZKmb28dQRO7Zc4QR-_qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true},
not same cluster_name [logger]

[2015-02-05 20:54:11,405][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_19_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]

[2015-02-05 20:54:11,406][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]

[2015-02-05 20:54:11,407][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]

[2015-02-05 20:54:11,408][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_20_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]

[2015-02-05 20:54:11,409][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_27_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]

[2015-02-05 20:54:11,413][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_21_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]

[2015-02-05 20:54:11,413][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_26_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]

[2015-02-05 20:54:11,413][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_25_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]

[2015-02-05 20:54:11,414][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]

[2015-02-05 20:54:11,414][TRACE][discovery.ec2 ] [d1b9ebf8c605] full ping
responses:

--> ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}

--> ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}

[2015-02-05 20:54:11,415][DEBUG][discovery.ec2 ] [d1b9ebf8c605] filtered
ping responses: (filter_client[true], filter_data[false])

--> ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}

--> ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}

[2015-02-05 20:54:11,419][TRACE][discovery.ec2 ] [d1b9ebf8c605] joining
master
[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}

[2015-02-05 20:54:11,434][DEBUG][discovery.zen.publish ] [d1b9ebf8c605]
received cluster state version 3

[2015-02-05 20:54:11,436][DEBUG][discovery.zen.fd ] [d1b9ebf8c605] [master]
restarting fault detection against master
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
reason [new cluster state received and we are monitoring the wrong master
[null]]

[2015-02-05 20:54:11,437][DEBUG][discovery.ec2 ] [d1b9ebf8c605] got first
state from fresh master [9nNJp0wHSi2sa44Cyk37Dw]

[2015-02-05 20:54:11,437][TRACE][discovery.ec2 ] [d1b9ebf8c605] updated
cluster join cluster to [1]

[2015-02-05 20:54:11,439][INFO ][cluster.service ] [d1b9ebf8c605]
detected_master
[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true},
added
{[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true},[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true},},
reason: zen-disco-receive(from master
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}])

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3a433856-2629-413f-a02f-0b9154302bb1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

dadoonet · February 6, 2015, 10:51am

What we do this in AWS plugin:

get the IP address from AWS API (which obviously does know anything about the port used)
address = instance.getPrivateIpAddress();
transform this String to a TransportAddress which will basically be IP:9300:
TransportAddress addresses = transportService.addressesFromString(address);
try to join IP:9300 which obviously answers but this is not the one you are expecting.

I wonder if setting transport.tcp.port: 9302 could help here.
By default this value is 9300-9399.

--
David Pilato | Technical Advocate | Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr https://twitter.com/elasticsearchfr | @scrutmydocs https://twitter.com/scrutmydocs

Le 6 févr. 2015 à 01:23, Ricardo Alves dos Reis ricardo.areis@gmail.com a écrit :

Hello folks,

First of all, I'd like to explain my current necessity to bring this future trend out, some projects in my company have bean developed to use a lot of micro-services, unfortunately, is not uncommon that the mean of cpu usage in our cloud servers (AWS) are sustained in a lower degrees.

To provisioning these resources easily all the components are executed in a docker containers and these stacks were represented as yaml file (http://www.fig.sh). In a group of servers I've more than one Elasticsearch clusters, containerized to use EC2 discovery and with a transport.publish_port configured to avoid collision, however, only the first cluster in default transport port have a correct cluster formation.

My guess is the transport.publish_post is not used as pointer to discovery.zen.unicast to avoid unused transport port..

Thanks for any kind of help.

docker #aws #discovery

fig.yml

esemail:
image: XXX/elasticsearch:latest
volumes:

/dev/log:/dev/log

/data0/email:/data/0

/data1/email:/data/1
ports:

"9202:9200"

"9302:9300"
environment:
TRANSPORT_PUBLISH_PORT: 9302
NETWORK_PUBLISH_HOST: ec2
CLUSTER_ROUTING_ALLOCATION_SAME_SHARD_HOST: "true"
ES_ROOT_LOG_LEVEL: INFO
ES_DISCOVERY_LOG_LEVEL: TRACE
ES_LOG_DIR: /var/log/elasticsearch
ES_HEAP_SIZE: 1g
ES_DIRECT_SIZE: 256m
ES_JAVA_OPTS: "-server -XX:+UseCompressedOops -XX:+DoEscapeAnalysis -XX:+AlwaysPreTouch -verbose:gc -Xloggc:/var/log/elasticsearch/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=512K"
MAX_OPEN_FILES: 65535
MAX_MAP_COUNT: 262144
CLUSTER_NAME: email-es
NODE_MASTER: "true"
NODE_DATA: "true"
PATH_DATA: /data/0/datasearch0,/data/1/datasearch0
BOOTSTRAP_MLOCKALL: "false"
DISCOVERY_ZEN_MCAST: "false"
DISCOVERY_ZEN_REJOIN_ON_MASTER_GONE: "false"
DISCOVERY_ZEN_MINIMUM_MASTER_NODES: 2
DISCOVERY_TYPE: ec2
DISCOVERY_EC2_ANY_GROUP: "false"
DISCOVERY_EC2_GROUPS: db-prod-es-mail
DISCOVERY_EC2_AVAILABILITY_ZONES: us-east-1a,us-east-1b,us-east-1d
DISCOVERY_EC2_HOST_TYPE: private_ip
DISCOVERY_EC2_PING_TIMEOUT: 1m
GATEWAY_RECOVER_AFTER_NODES: 2
GATEWAY_RECOVER_AFTER_TIME: 2m
GATEWAY_EXPECTED_NODES: 3
AWS_ELB_NAME: ESEmailBionexo
AWS_ACCESS_KEY_ID: XXXXX
AWS_SECRET_ACCESS_KEY: XXXX
AWS_DEFAULT_REGION: us-east-1
THREAD_POOL_SEARCH: 2000
THREAD_POOL_INDEX: 2000
THREAD_POOL_BULK: 2000

[ec2-user@ip-172-16-255-20 ~]$ sudo docker ps |grep elastic
d1b9ebf8c605 bionexo/elasticsearch:2015020502 "/bin/sh -c /bionexo 3 hours ago Up 3 hours 0.0.0.0:9202->9200/tcp, 0.0.0.0:9302->9300/tcp elasticsearch_esemail_1
088da31f943b bionexo/elasticsearch:2015020401 "/bin/sh -c /bionexo 35 hours ago Up 35 hours 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp elasticsearch_eslogger_1

[2015-02-05 20:53:09,834][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:09,853][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:09,854][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:10,637][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:10,637][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:23,833][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:25,837][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:26,618][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:35,724][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:38,034][TRACE][discovery.ec2 ] [d1b9ebf8c605] building dynamic unicast discovery nodes...
[2015-02-05 20:53:38,034][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-713a399d based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,034][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-7bd2f097 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-0b25abfa based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-83c39ca2 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: automation,GroupId: sg-90c510f5}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-7d49e897 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-fb9cc0d7 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-a72ef14d based on groups [{GroupName: corporativo,GroupId: sg-8f9c46ea}, {GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-54763907 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-b98e5e48 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,036][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding i-b0384441, address 172.16.250.83, transport_address inet[/172.16.250.83:9300]
[2015-02-05 20:53:38,036][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-84b324d7 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: demo_apps,GroupId: sg-44c31d21}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,036][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-eb29db01 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-6cdd259d based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-53cabead based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-f5f21004 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-fd21a317 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-ebd815bb based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-81187b70 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,038][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding i-4d52afb7, address 172.16.255.20, transport_address inet[/172.16.255.20:9300]
[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding i-90f2367f, address 172.16.255.149, transport_address inet[/172.16.255.149:9300]
[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-b25d8e5f based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-4517fab4 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-573bf607 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: vpn,GroupId: sg-4fac3e2a}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-ca09f627 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: varnish,GroupId: sg-3815eb5c}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-a2728b53 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-fb222600 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-502433bc based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-2e871305 based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-3a7f91c0 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-19a469e3 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-e8038e04 based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,040][DEBUG][discovery.ec2 ] [d1b9ebf8c605] using dynamic discovery nodes [[#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]], [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]], [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]]
[2015-02-05 20:53:38,041][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] replacing [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]] with temp node [#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connecting (light) to [#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] replacing [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]] with temp node [#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connecting (light) to [#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] replacing [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]] with temp node [#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connected to [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]], using >=1.4.0 serialization
[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connecting (light) to [#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:53:38,044][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connected to [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]], using >=1.4.0 serialization
[2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connected to [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]], using >=1.4.0 serialization
[2015-02-05 20:53:38,046][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}
[2015-02-05 20:53:38,055][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]: [ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[5], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true}], id[442], master [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], hasJoinedOnce [true], cluster_name[logger]}]
[2015-02-05 20:53:38,054][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]: [ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[5], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], id[454], master [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], hasJoinedOnce [true], cluster_name[logger]}]
[2015-02-05 20:53:38,055][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] filtering out response from [088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}, not same cluster_name [logger]
[2015-02-05 20:53:38,056][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] filtering out response from [1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true}, not same cluster_name [logger]
[2015-02-05 20:53:38,057][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}: [ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[5], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[16], master [null], hasJoinedOnce [false], cluster_name[email-es]}]
[2015-02-05 20:53:38,857][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:38,864][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:38,864][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:38,866][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:38,886][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:38,890][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,659][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,660][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,665][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,674][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,714][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,719][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,722][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,744][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,851][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:39,855][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,854][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:39,853][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:39,857][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:39,859][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,927][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,929][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,929][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,931][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,931][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,934][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,934][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,935][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,937][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,937][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:39,938][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,037][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,038][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,039][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,044][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,044][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,046][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,047][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,048][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,048][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,049][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,049][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,050][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,053][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,054][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,068][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,071][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,074][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,079][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,080][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,082][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,619][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,622][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,623][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,632][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,632][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,634][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,636][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,644][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,647][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,657][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,658][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,662][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,671][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,671][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,671][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,675][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:40,684][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,686][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,687][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,687][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,689][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,689][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,690][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,691][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,692][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,692][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,692][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,694][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,692][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,695][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,694][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,700][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,700][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,985][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:41,172][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,177][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,182][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,183][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,186][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,188][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,194][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,197][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,200][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,201][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,203][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,203][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,207][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,208][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,210][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,211][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,215][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,218][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,229][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,231][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,234][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,236][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,240][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,241][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,241][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,242][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:42,249][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:43,617][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:44,206][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:45,041][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]: [ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[5], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[22b068e7698f][HZKmb28dQRO7Zc4QR-qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true}], id[43], master [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], hasJoinedOnce [true], cluster_name[logger]}]
[2015-02-05 20:53:45,041][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] filtering out response from [22b068e7698f][HZKmb28dQRO7Zc4QR-qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true}, not same cluster_name [logger]
[2015-02-05 20:53:45,288][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:46,241][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:48,549][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:48,808][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:49,835][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:51,279][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:52,079][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:53,051][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:53,359][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:54,618][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:54,621][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:54,850][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:57,485][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:53:58,526][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:59,842][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:54:00,938][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:54:01,047][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:54:02,304][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:54:03,105][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:54:04,315][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:54:06,436][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:54:08,295][TRACE][discovery.ec2 ] [d1b9ebf8c605] building dynamic unicast discovery nodes...
[2015-02-05 20:54:08,297][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-713a399d based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,297][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-7bd2f097 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,297][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-0b25abfa based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,297][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-83c39ca2 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: automation,GroupId: sg-90c510f5}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,297][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-7d49e897 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,298][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-fb9cc0d7 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,298][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-a72ef14d based on groups [{GroupName: corporativo,GroupId: sg-8f9c46ea}, {GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,298][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-54763907 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,299][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-b98e5e48 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,301][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding i-b0384441, address 172.16.250.83, transport_address inet[/172.16.250.83:9300]
[2015-02-05 20:54:08,301][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-84b324d7 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: demo_apps,GroupId: sg-44c31d21}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,301][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-eb29db01 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,301][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-6cdd259d based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,301][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-53cabead based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,302][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-f5f21004 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,302][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-fd21a317 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,302][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-ebd815bb based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,302][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-81187b70 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,304][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding i-4d52afb7, address 172.16.255.20, transport_address inet[/172.16.255.20:9300]
[2015-02-05 20:54:08,312][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding i-90f2367f, address 172.16.255.149, transport_address inet[/172.16.255.149:9300]
[2015-02-05 20:54:08,313][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-b25d8e5f based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,313][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-4517fab4 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,313][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-573bf607 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: vpn,GroupId: sg-4fac3e2a}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,315][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-ca09f627 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: varnish,GroupId: sg-3815eb5c}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-a2728b53 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-fb222600 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-502433bc based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName: host-healthcheck,GroupId: sg-73df7e17}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-2e871305 based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-3a7f91c0 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-19a469e3 based on groups [{GroupName: default,GroupId: sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,317][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering out instance i-e8038e04 based on groups [{GroupName: default,GroupId: sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,317][DEBUG][discovery.ec2 ] [d1b9ebf8c605] using dynamic discovery nodes [[#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]], [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]], [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]]
[2015-02-05 20:54:08,317][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] replacing [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]] with temp node [#zen_unicast_25#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:54:08,317][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connecting (light) to [#zen_unicast_25#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:54:08,318][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] replacing [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]] with temp node [#zen_unicast_26_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:54:08,319][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connected to [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:54:08,319][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [#zen_unicast_25_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]], using >=1.4.0 serialization
[2015-02-05 20:54:08,322][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [#zen_unicast_25_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]: [ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], id[457], master [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], hasJoinedOnce [true], cluster_name[logger]}]
[2015-02-05 20:54:08,322][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] filtering out response from [088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}, not same cluster_name [logger]
[2015-02-05 20:54:08,325][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connecting (light) to [#zen_unicast_26_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:54:08,326][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] replacing [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]] with temp node [#zen_unicast_27_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:54:08,327][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connecting (light) to [#zen_unicast_27_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:54:08,327][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}
[2015-02-05 20:54:08,328][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connected to [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:54:08,334][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [#zen_unicast_27_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]], using >=1.4.0 serialization
[2015-02-05 20:54:08,328][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] connected to [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:54:08,334][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] sending to [#zen_unicast_26_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]], using >=1.4.0 serialization
[2015-02-05 20:54:08,341][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [#zen_unicast_26_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]: [ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true}], id[445], master [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], hasJoinedOnce [true], cluster_name[logger]}]
[2015-02-05 20:54:08,341][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] filtering out response from [1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true}, not same cluster_name [logger]
[2015-02-05 20:54:08,342][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}: [ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[18], master [null], hasJoinedOnce [false], cluster_name[email-es]}]
[2015-02-05 20:54:09,814][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:54:09,835][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:54:10,616][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:54:10,637][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:54:10,638][DEBUG][action.admin.indices.create] [d1b9ebf8c605] observer: timeout notification from cluster service. timeout setting [1m], time since start [1m]
[2015-02-05 20:54:10,854][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:54:10,857][DEBUG][action.admin.indices.create] [d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:54:11,404][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] received response from [#zen_unicast_27_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]: [ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}, ping_response{node [[22b068e7698f][HZKmb28dQRO7Zc4QR-qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true}], id[46], master [[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}], hasJoinedOnce [true], cluster_name[logger]}]
[2015-02-05 20:54:11,405][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] filtering out response from [22b068e7698f][HZKmb28dQRO7Zc4QR-qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true}, not same cluster_name [logger]
[2015-02-05 20:54:11,405][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen_unicast_19#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:54:11,406][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen_unicast_22#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:54:11,407][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:54:11,408][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen_unicast_20_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:54:11,409][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen_unicast_27_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:54:11,413][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen_unicast_21_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:54:11,413][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen_unicast_26_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:54:11,413][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen_unicast_25_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:54:11,414][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605] [3] disconnecting from [#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:54:11,414][TRACE][discovery.ec2 ] [d1b9ebf8c605] full ping responses:
--> ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}
--> ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}
[2015-02-05 20:54:11,415][DEBUG][discovery.ec2 ] [d1b9ebf8c605] filtered ping responses: (filter_client[true], filter_data[false])
--> ping_response{node [[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}
--> ping_response{node [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}
[2015-02-05 20:54:11,419][TRACE][discovery.ec2 ] [d1b9ebf8c605] joining master [d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}
[2015-02-05 20:54:11,434][DEBUG][discovery.zen.publish ] [d1b9ebf8c605] received cluster state version 3
[2015-02-05 20:54:11,436][DEBUG][discovery.zen.fd ] [d1b9ebf8c605] [master] restarting fault detection against master [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}], reason [new cluster state received and we are monitoring the wrong master [null]]
[2015-02-05 20:54:11,437][DEBUG][discovery.ec2 ] [d1b9ebf8c605] got first state from fresh master [9nNJp0wHSi2sa44Cyk37Dw]
[2015-02-05 20:54:11,437][TRACE][discovery.ec2 ] [d1b9ebf8c605] updated cluster join cluster to [1]
[2015-02-05 20:54:11,439][INFO ][cluster.service ] [d1b9ebf8c605] detected_master [d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}, added {[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true},[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true},}, reason: zen-disco-receive(from master [[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}])

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3a433856-2629-413f-a02f-0b9154302bb1%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/3a433856-2629-413f-a02f-0b9154302bb1%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/091AE197-22B3-495A-832B-757941F6A8D4%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Ricardo_Reis · February 12, 2015, 1:33pm

Hello David,

It's really solve my problem

Thanks

Ricardo Reis

Em sexta-feira, 6 de fevereiro de 2015 08:52:03 UTC-2, David Pilato
escreveu:

What we do this in AWS plugin:

get the IP address from AWS API (which obviously does know anything
about the port used)
address = instance.getPrivateIpAddress();

transform this String to a TransportAddress which will basically be
IP:9300:
TransportAddress addresses
= transportService.addressesFromString(address);

try to join IP:9300 which obviously answers but this is not the one you
are expecting.

I wonder if setting transport.tcp.port: 9302 could help here.
By default this value is 9300-9399.

--
David Pilato | Technical Advocate | Elasticsearch.com
http://Elasticsearch.com
@dadoonet https://twitter.com/dadoonet | @elasticsearchfr
https://twitter.com/elasticsearchfr | @scrutmydocs
https://twitter.com/scrutmydocs

Le 6 févr. 2015 à 01:23, Ricardo Alves dos Reis <ricard...@gmail.com
<javascript:>> a écrit :

Hello folks,

First of all, I'd like to explain my current necessity to bring this
future trend out, some projects in my company have bean developed to use a
lot of micro-services, unfortunately, is not uncommon that the mean of cpu
usage in our cloud servers (AWS) are sustained in a lower degrees.

To provisioning these resources easily all the components are executed in
a docker containers and these stacks were represented as yaml file (
http://www.fig.sh). In a group of servers I've more than one
Elasticsearch clusters, containerized to use EC2 discovery and with a
transport.publish_port configured to avoid collision, however, only the
first cluster in default transport port have a correct cluster formation.

My guess is the transport.publish_post is not used as pointer to
discovery.zen.unicast to avoid unused transport port..

Thanks for any kind of help.

docker #aws #discovery

fig.yml

esemail:

image: XXX/elasticsearch:latest

volumes:

/dev/log:/dev/log

/data0/email:/data/0

/data1/email:/data/1

ports:

"9202:9200"

"9302:9300"

environment:

TRANSPORT_PUBLISH_PORT: 9302

NETWORK_PUBLISH_HOST: ec2

CLUSTER_ROUTING_ALLOCATION_SAME_SHARD_HOST: "true"

ES_ROOT_LOG_LEVEL: INFO

ES_DISCOVERY_LOG_LEVEL: TRACE

ES_LOG_DIR: /var/log/elasticsearch

ES_HEAP_SIZE: 1g

ES_DIRECT_SIZE: 256m

ES_JAVA_OPTS: "-server -XX:+UseCompressedOops -XX:+DoEscapeAnalysis
-XX:+AlwaysPreTouch -verbose:gc -Xloggc:/var/log/elasticsearch/gc.log
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=512K"

MAX_OPEN_FILES: 65535

MAX_MAP_COUNT: 262144

CLUSTER_NAME: email-es

NODE_MASTER: "true"

NODE_DATA: "true"

PATH_DATA: /data/0/datasearch0,/data/1/datasearch0

BOOTSTRAP_MLOCKALL: "false"

DISCOVERY_ZEN_MCAST: "false"

DISCOVERY_ZEN_REJOIN_ON_MASTER_GONE: "false"

DISCOVERY_ZEN_MINIMUM_MASTER_NODES: 2

DISCOVERY_TYPE: ec2

DISCOVERY_EC2_ANY_GROUP: "false"

DISCOVERY_EC2_GROUPS: db-prod-es-mail

DISCOVERY_EC2_AVAILABILITY_ZONES: us-east-1a,us-east-1b,us-east-1d

DISCOVERY_EC2_HOST_TYPE: private_ip

DISCOVERY_EC2_PING_TIMEOUT: 1m

GATEWAY_RECOVER_AFTER_NODES: 2

GATEWAY_RECOVER_AFTER_TIME: 2m

GATEWAY_EXPECTED_NODES: 3

AWS_ELB_NAME: ESEmailBionexo

AWS_ACCESS_KEY_ID: XXXXX

AWS_SECRET_ACCESS_KEY: XXXX

AWS_DEFAULT_REGION: us-east-1

THREAD_POOL_SEARCH: 2000

THREAD_POOL_INDEX: 2000

THREAD_POOL_BULK: 2000

[ec2-user@ip-172-16-255-20 ~]$ sudo docker ps |grep elastic

d1b9ebf8c605 bionexo/elasticsearch:2015020502 "/bin/sh -c /bionexo 3 hours
ago Up 3 hours 0.0.0.0:9202->9200/tcp, 0.0.0.0:9302->9300/tcp
elasticsearch_esemail_1

088da31f943b bionexo/elasticsearch:2015020401 "/bin/sh -c /bionexo 35
hours ago Up 35 hours 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp
elasticsearch_eslogger_1

[2015-02-05 20:53:09,834][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:09,853][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:09,854][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:10,637][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:10,637][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:23,833][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:25,837][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:26,618][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry

[2015-02-05 20:53:35,724][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]

[2015-02-05 20:53:38,034][TRACE][discovery.ec2 ] [d1b9ebf8c605] building
dynamic unicast discovery nodes...

[2015-02-05 20:53:38,034][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-713a399d based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,034][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-7bd2f097 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-0b25abfa based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-83c39ca2 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: automation,GroupId: sg-90c510f5}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-7d49e897 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-fb9cc0d7 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-a72ef14d based on groups [{GroupName: corporativo,GroupId:
sg-8f9c46ea}, {GroupName: default,GroupId: sg-62e42407}], does not include
all of [db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-54763907 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-b98e5e48 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,036][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding
i-b0384441, address 172.16.250.83, transport_address inet[/
172.16.250.83:9300]

[2015-02-05 20:53:38,036][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-84b324d7 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: demo_apps,GroupId: sg-44c31d21}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,036][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-eb29db01 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-6cdd259d based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-53cabead based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]

[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-f5f21004 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-fd21a317 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-ebd815bb based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-81187b70 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,038][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding
i-4d52afb7, address 172.16.255.20, transport_address inet[/
172.16.255.20:9300]

[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding
i-90f2367f, address 172.16.255.149, transport_address inet[/
172.16.255.149:9300]

[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-b25d8e5f based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-4517fab4 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-573bf607 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: vpn,GroupId: sg-4fac3e2a}], does not include all
of [db-prod-es-mail]

[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-ca09f627 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: varnish,GroupId: sg-3815eb5c}], does not include
all of [db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-a2728b53 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-fb222600 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-502433bc based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-2e871305 based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-3a7f91c0 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-19a469e3 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]

[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-e8038e04 based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]

[2015-02-05 20:53:38,040][DEBUG][discovery.ec2 ] [d1b9ebf8c605] using
dynamic discovery nodes
[[#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]],
[#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]],
[#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]]

[2015-02-05 20:53:38,041][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605] replacing [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/
172.16.255.20:9300]] with temp node
[#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/
172.16.255.20:9300]]

[2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605] [3] connecting (light) to
[#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/
172.16.255.20:9300]]

[2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605] replacing [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/
172.16.255.149:9300]] with temp node
[#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/
172.16.255.149:9300]]

[2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605] [3] connecting (light) to
[#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/
172.16.255.149:9300]]

[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605] replacing [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/
172.16.250.83:9300]] with temp node
[#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/
172.16.250.83:9300]]

[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605] [3] connected to [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/
172.16.255.20:9300]]

[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605] [3] sending to
[#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]],
using >=1.4.0 serialization

[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605] [3] connecting (light) to
[#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/
172.16.250.83:9300]]

[2015-02-05 20:53:38,044][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605] [3] connected to [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/
172.16.255.149:9300]]

[2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605] [3] sending to
[#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]],
using >=1.4.0 serialization

[2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605] [3] connected to [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/
172.16.250.83:9300]]

[2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605] [3] sending to
[#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]],
using >=1.4.0 serialization

[2015-02-05 20:53:38,046][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605] [3] sending to
[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}

[2015-02-05 20:53:38,055][TRACE][discovery.zen.ping.unicast]
[d1b9ebf8c605]

...

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/92322f0d-c420-48f0-baf7-fa69934ef1d1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Topic		Replies	Views
Overriding `tcp.publish_port` breaks clustering when elasticsearch is in a container Elasticsearch	4	1340	July 5, 2017
Multi elasticsearch docker containers per host? Elasticsearch	3	5426	July 5, 2017
Deploy Elastic Search cluster consisting of multiple hosts using ES docker image Elasticsearch	2	1288	August 4, 2018
Hosting multiple ES instances on one host Elasticsearch	10	9564	July 5, 2017
Clustering Dockerized Elasticsearch with multiple Docker Host Elasticsearch	1	2183	September 3, 2019

Different ES clusters in the same host

Related topics