Hello folks,
First of all, I'd like to explain my current necessity to bring this
future trend out, some projects in my company have bean developed to use a
lot of micro-services, unfortunately, is not uncommon that the mean of cpu
usage in our cloud servers (AWS) are sustained in a lower degrees.
To provisioning these resources easily all the components are executed in
a docker containers and these stacks were represented as yaml file
(http://www.fig.sh). In a group of servers I've more than one Elasticsearch
clusters, containerized to use EC2 discovery and with a
transport.publish_port configured to avoid collision, however, only the
first cluster in default transport port have a correct cluster formation.
My guess is the transport.publish_post is not used as pointer to
discovery.zen.unicast to avoid unused transport port..
Thanks for any kind of help.
#docker #aws #discovery
fig.yml
esemail:
image: XXX/elasticsearch:latest
volumes:
-
/dev/log:/dev/log
-
/data0/email:/data/0
-
/data1/email:/data/1
ports:
-
"9202:9200"
-
"9302:9300"
environment:
TRANSPORT_PUBLISH_PORT: 9302
NETWORK_PUBLISH_HOST: ec2
CLUSTER_ROUTING_ALLOCATION_SAME_SHARD_HOST: "true"
ES_ROOT_LOG_LEVEL: INFO
ES_DISCOVERY_LOG_LEVEL: TRACE
ES_LOG_DIR: /var/log/elasticsearch
ES_HEAP_SIZE: 1g
ES_DIRECT_SIZE: 256m
ES_JAVA_OPTS: "-server -XX:+UseCompressedOops -XX:+DoEscapeAnalysis
-XX:+AlwaysPreTouch -verbose:gc -Xloggc:/var/log/elasticsearch/gc.log
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=512K"
MAX_OPEN_FILES: 65535
MAX_MAP_COUNT: 262144
CLUSTER_NAME: email-es
NODE_MASTER: "true"
NODE_DATA: "true"
PATH_DATA: /data/0/datasearch0,/data/1/datasearch0
BOOTSTRAP_MLOCKALL: "false"
DISCOVERY_ZEN_MCAST: "false"
DISCOVERY_ZEN_REJOIN_ON_MASTER_GONE: "false"
DISCOVERY_ZEN_MINIMUM_MASTER_NODES: 2
DISCOVERY_TYPE: ec2
DISCOVERY_EC2_ANY_GROUP: "false"
DISCOVERY_EC2_GROUPS: db-prod-es-mail
DISCOVERY_EC2_AVAILABILITY_ZONES: us-east-1a,us-east-1b,us-east-1d
DISCOVERY_EC2_HOST_TYPE: private_ip
DISCOVERY_EC2_PING_TIMEOUT: 1m
GATEWAY_RECOVER_AFTER_NODES: 2
GATEWAY_RECOVER_AFTER_TIME: 2m
GATEWAY_EXPECTED_NODES: 3
AWS_ELB_NAME: ESEmailBionexo
AWS_ACCESS_KEY_ID: XXXXX
AWS_SECRET_ACCESS_KEY: XXXX
AWS_DEFAULT_REGION: us-east-1
THREAD_POOL_SEARCH: 2000
THREAD_POOL_INDEX: 2000
THREAD_POOL_BULK: 2000
[ec2-user@ip-172-16-255-20 ~]$ sudo docker ps |grep elastic
d1b9ebf8c605 bionexo/elasticsearch:2015020502 "/bin/sh -c /bionexo 3 hours
ago Up 3 hours 0.0.0.0:9202->9200/tcp, 0.0.0.0:9302->9300/tcp
elasticsearch_esemail_1
088da31f943b bionexo/elasticsearch:2015020401 "/bin/sh -c /bionexo 35 hours
ago Up 35 hours 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp
elasticsearch_eslogger_1
[2015-02-05 20:53:09,834][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:09,853][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:09,854][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:10,637][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:10,637][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:23,833][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:25,837][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:26,618][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:35,724][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:38,034][TRACE][discovery.ec2 ] [d1b9ebf8c605] building
dynamic unicast discovery nodes...
[2015-02-05 20:53:38,034][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-713a399d based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]
[2015-02-05 20:53:38,034][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-7bd2f097 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]
[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-0b25abfa based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-83c39ca2 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: automation,GroupId: sg-90c510f5}], does not
include all of [db-prod-es-mail]
[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-7d49e897 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]
[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-fb9cc0d7 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]
[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-a72ef14d based on groups [{GroupName: corporativo,GroupId:
sg-8f9c46ea}, {GroupName: default,GroupId: sg-62e42407}], does not include
all of [db-prod-es-mail]
[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-54763907 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]
[2015-02-05 20:53:38,035][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-b98e5e48 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]
[2015-02-05 20:53:38,036][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding
i-b0384441, address 172.16.250.83, transport_address
inet[/172.16.250.83:9300]
[2015-02-05 20:53:38,036][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-84b324d7 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: demo_apps,GroupId: sg-44c31d21}], does not
include all of [db-prod-es-mail]
[2015-02-05 20:53:38,036][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-eb29db01 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]
[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-6cdd259d based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]
[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-53cabead based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-f5f21004 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]
[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-fd21a317 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]
[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-ebd815bb based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]
[2015-02-05 20:53:38,037][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-81187b70 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]
[2015-02-05 20:53:38,038][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding
i-4d52afb7, address 172.16.255.20, transport_address
inet[/172.16.255.20:9300]
[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding
i-90f2367f, address 172.16.255.149, transport_address
inet[/172.16.255.149:9300]
[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-b25d8e5f based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]
[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-4517fab4 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]
[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-573bf607 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: vpn,GroupId: sg-4fac3e2a}], does not include all
of [db-prod-es-mail]
[2015-02-05 20:53:38,039][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-ca09f627 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: varnish,GroupId: sg-3815eb5c}], does not include
all of [db-prod-es-mail]
[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-a2728b53 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]
[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-fb222600 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]
[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-502433bc based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]
[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-2e871305 based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-3a7f91c0 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]
[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-19a469e3 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]
[2015-02-05 20:53:38,040][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-e8038e04 based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:53:38,040][DEBUG][discovery.ec2 ] [d1b9ebf8c605] using
dynamic discovery nodes
[[#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]],
[#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]],
[#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]]
[2015-02-05 20:53:38,041][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
replacing [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]]
with temp node
[#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connecting (light) to
[#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
replacing [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]
with temp node
[#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:53:38,042][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connecting (light) to
[#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
replacing [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]]
with temp node
[#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connected to
[#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] sending to
[#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]],
using >=1.4.0 serialization
[2015-02-05 20:53:38,043][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connecting (light) to
[#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:53:38,044][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connected to
[#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] sending to
[#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]],
using >=1.4.0 serialization
[2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connected to
[#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:53:38,045][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] sending to
[#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]],
using >=1.4.0 serialization
[2015-02-05 20:53:38,046][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] sending to
[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}
[2015-02-05 20:53:38,055][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] received response from
[#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]:
[ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[5], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true}],
id[442], master
[[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}],
hasJoinedOnce [true], cluster_name[logger]}]
[2015-02-05 20:53:38,054][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] received response from
[#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]:
[ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[5], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}],
id[454], master
[[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}],
hasJoinedOnce [true], cluster_name[logger]}]
[2015-02-05 20:53:38,055][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] filtering out response from
[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true},
not same cluster_name [logger]
[2015-02-05 20:53:38,056][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] filtering out response from
[1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true},
not same cluster_name [logger]
[2015-02-05 20:53:38,057][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] received response from
[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}:
[ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[5], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[16], master [null], hasJoinedOnce [false], cluster_name[email-es]}]
[2015-02-05 20:53:38,857][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:38,864][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:38,864][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:38,866][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:38,886][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:38,890][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:39,659][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:39,660][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:39,665][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:39,674][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:39,714][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:39,719][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:39,722][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:39,744][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:39,851][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:39,855][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:39,854][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:39,853][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:39,857][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:39,859][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:39,927][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:39,929][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:39,929][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:39,931][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:39,931][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:39,934][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:39,934][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:39,935][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:39,937][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:39,937][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:39,938][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,037][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,038][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,039][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,044][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,044][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,046][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,047][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,048][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,048][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,049][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,049][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,050][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,053][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,054][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,068][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,071][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,074][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,079][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,080][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,082][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,619][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,622][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,623][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,632][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,632][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,634][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,636][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,644][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,647][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,657][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,658][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,662][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,671][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,671][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,671][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,675][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:40,684][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,686][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,687][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,687][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,689][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,689][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,690][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,691][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,692][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,692][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,692][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,694][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,692][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,695][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,694][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,700][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,700][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:40,985][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:41,172][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,177][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,182][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,183][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,186][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,188][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,194][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,197][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,200][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,201][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,203][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,203][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,207][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,208][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,210][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,211][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,215][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,218][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,229][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,231][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,234][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,236][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,240][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,241][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,241][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:41,242][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:42,249][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:43,617][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:44,206][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:45,041][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] received response from
[#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]:
[ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[5], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[7], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[22b068e7698f][HZKmb28dQRO7Zc4QR-_qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true}],
id[43], master
[[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}],
hasJoinedOnce [true], cluster_name[logger]}]
[2015-02-05 20:53:45,041][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] filtering out response from
[22b068e7698f][HZKmb28dQRO7Zc4QR-_qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true},
not same cluster_name [logger]
[2015-02-05 20:53:45,288][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:46,241][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:48,549][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:48,808][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:49,835][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:51,279][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:52,079][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:53,051][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:53,359][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:54,618][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:54,621][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:54,850][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:57,485][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:53:58,526][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:53:59,842][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:54:00,938][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:54:01,047][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:54:02,304][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:54:03,105][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:54:04,315][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:54:06,436][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:54:08,295][TRACE][discovery.ec2 ] [d1b9ebf8c605] building
dynamic unicast discovery nodes...
[2015-02-05 20:54:08,297][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-713a399d based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]
[2015-02-05 20:54:08,297][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-7bd2f097 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]
[2015-02-05 20:54:08,297][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-0b25abfa based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,297][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-83c39ca2 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: automation,GroupId: sg-90c510f5}], does not
include all of [db-prod-es-mail]
[2015-02-05 20:54:08,297][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-7d49e897 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]
[2015-02-05 20:54:08,298][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-fb9cc0d7 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]
[2015-02-05 20:54:08,298][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-a72ef14d based on groups [{GroupName: corporativo,GroupId:
sg-8f9c46ea}, {GroupName: default,GroupId: sg-62e42407}], does not include
all of [db-prod-es-mail]
[2015-02-05 20:54:08,298][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-54763907 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]
[2015-02-05 20:54:08,299][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-b98e5e48 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]
[2015-02-05 20:54:08,301][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding
i-b0384441, address 172.16.250.83, transport_address
inet[/172.16.250.83:9300]
[2015-02-05 20:54:08,301][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-84b324d7 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: demo_apps,GroupId: sg-44c31d21}], does not
include all of [db-prod-es-mail]
[2015-02-05 20:54:08,301][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-eb29db01 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]
[2015-02-05 20:54:08,301][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-6cdd259d based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]
[2015-02-05 20:54:08,301][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-53cabead based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,302][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-f5f21004 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]
[2015-02-05 20:54:08,302][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-fd21a317 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}], does not
include all of [db-prod-es-mail]
[2015-02-05 20:54:08,302][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-ebd815bb based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]
[2015-02-05 20:54:08,302][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-81187b70 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]
[2015-02-05 20:54:08,304][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding
i-4d52afb7, address 172.16.255.20, transport_address
inet[/172.16.255.20:9300]
[2015-02-05 20:54:08,312][TRACE][discovery.ec2 ] [d1b9ebf8c605] adding
i-90f2367f, address 172.16.255.149, transport_address
inet[/172.16.255.149:9300]
[2015-02-05 20:54:08,313][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-b25d8e5f based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]
[2015-02-05 20:54:08,313][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-4517fab4 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]
[2015-02-05 20:54:08,313][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-573bf607 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: vpn,GroupId: sg-4fac3e2a}], does not include all
of [db-prod-es-mail]
[2015-02-05 20:54:08,315][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-ca09f627 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: varnish,GroupId: sg-3815eb5c}], does not include
all of [db-prod-es-mail]
[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-a2728b53 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]
[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-fb222600 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]
[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-502433bc based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: prod_apps,GroupId: sg-620d6b07}, {GroupName:
host-healthcheck,GroupId: sg-73df7e17}], does not include all of
[db-prod-es-mail]
[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-2e871305 based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-3a7f91c0 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]
[2015-02-05 20:54:08,316][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-19a469e3 based on groups [{GroupName: default,GroupId:
sg-62e42407}, {GroupName: db-prod-oracle,GroupId: sg-f972d79c}], does not
include all of [db-prod-es-mail]
[2015-02-05 20:54:08,317][TRACE][discovery.ec2 ] [d1b9ebf8c605] filtering
out instance i-e8038e04 based on groups [{GroupName: default,GroupId:
sg-62e42407}], does not include all of [db-prod-es-mail]
[2015-02-05 20:54:08,317][DEBUG][discovery.ec2 ] [d1b9ebf8c605] using
dynamic discovery nodes
[[#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]],
[#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]],
[#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]]
[2015-02-05 20:54:08,317][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
replacing [#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]]
with temp node
[#zen_unicast_25_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:54:08,317][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connecting (light) to
[#zen_unicast_25_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:54:08,318][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
replacing [#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]
with temp node
[#zen_unicast_26_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:54:08,319][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connected to
[#cloud-i-4d52afb7-0][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:54:08,319][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] sending to
[#zen_unicast_25_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]],
using >=1.4.0 serialization
[2015-02-05 20:54:08,322][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] received response from
[#zen_unicast_25_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]:
[ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}],
id[457], master
[[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}],
hasJoinedOnce [true], cluster_name[logger]}]
[2015-02-05 20:54:08,322][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] filtering out response from
[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true},
not same cluster_name [logger]
[2015-02-05 20:54:08,325][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connecting (light) to
[#zen_unicast_26_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:54:08,326][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
replacing [#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]]
with temp node
[#zen_unicast_27_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:54:08,327][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connecting (light) to
[#zen_unicast_27_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:54:08,327][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] sending to
[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}
[2015-02-05 20:54:08,328][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connected to
[#cloud-i-b0384441-0][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:54:08,334][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] sending to
[#zen_unicast_27_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]],
using >=1.4.0 serialization
[2015-02-05 20:54:08,328][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] connected to
[#cloud-i-90f2367f-0][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:54:08,334][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] sending to
[#zen_unicast_26_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]],
using >=1.4.0 serialization
[2015-02-05 20:54:08,341][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] received response from
[#zen_unicast_26_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]:
[ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true}],
id[445], master
[[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}],
hasJoinedOnce [true], cluster_name[logger]}]
[2015-02-05 20:54:08,341][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] filtering out response from
[1e137c0938c3][BJO6p-PWQVyKlT8ycjNl7Q][1e137c0938c3][inet[/172.16.255.149:9300]]{master=true},
not same cluster_name [logger]
[2015-02-05 20:54:08,342][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] received response from
[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}:
[ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[18], master [null], hasJoinedOnce [false], cluster_name[email-es]}]
[2015-02-05 20:54:09,814][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:54:09,835][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:54:10,616][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:54:10,637][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:54:10,638][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] observer: timeout notification from cluster service. timeout
setting [1m], time since start [1m]
[2015-02-05 20:54:10,854][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:54:10,857][DEBUG][action.admin.indices.create]
[d1b9ebf8c605] no known master node, scheduling a retry
[2015-02-05 20:54:11,404][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] received response from
[#zen_unicast_27_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]:
[ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[9], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[11], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[13], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[15], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[d1b9ebf8c605][KT-HWB_xRpWhis_7TceU1A][d1b9ebf8c605][inet[/172.16.255.20:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]},
ping_response{node
[[22b068e7698f][HZKmb28dQRO7Zc4QR-_qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true}],
id[46], master
[[088da31f943b][Z2o4zFD8Q8q1fciYEExR2g][088da31f943b][inet[/172.16.255.20:9300]]{master=true}],
hasJoinedOnce [true], cluster_name[logger]}]
[2015-02-05 20:54:11,405][DEBUG][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] filtering out response from
[22b068e7698f][HZKmb28dQRO7Zc4QR-_qxg][22b068e7698f][inet[/172.16.250.83:9300]]{master=true},
not same cluster_name [logger]
[2015-02-05 20:54:11,405][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_19_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:54:11,406][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_22_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:54:11,407][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_24_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:54:11,408][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_20_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:54:11,409][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_27_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:54:11,413][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_21_#cloud-i-b0384441-0#][d1b9ebf8c605][inet[/172.16.250.83:9300]]
[2015-02-05 20:54:11,413][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_26_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:54:11,413][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_25_#cloud-i-4d52afb7-0#][d1b9ebf8c605][inet[/172.16.255.20:9300]]
[2015-02-05 20:54:11,414][TRACE][discovery.zen.ping.unicast] [d1b9ebf8c605]
[3] disconnecting from
[#zen_unicast_23_#cloud-i-90f2367f-0#][d1b9ebf8c605][inet[/172.16.255.149:9300]]
[2015-02-05 20:54:11,414][TRACE][discovery.ec2 ] [d1b9ebf8c605] full ping
responses:
--> ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}
--> ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}
[2015-02-05 20:54:11,415][DEBUG][discovery.ec2 ] [d1b9ebf8c605] filtered
ping responses: (filter_client[true], filter_data[false])
--> ping_response{node
[[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}
--> ping_response{node
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
id[17], master [null], hasJoinedOnce [false], cluster_name[email-es]}
[2015-02-05 20:54:11,419][TRACE][discovery.ec2 ] [d1b9ebf8c605] joining
master
[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}
[2015-02-05 20:54:11,434][DEBUG][discovery.zen.publish ] [d1b9ebf8c605]
received cluster state version 3
[2015-02-05 20:54:11,436][DEBUG][discovery.zen.fd ] [d1b9ebf8c605] [master]
restarting fault detection against master
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}],
reason [new cluster state received and we are monitoring the wrong master
[null]]
[2015-02-05 20:54:11,437][DEBUG][discovery.ec2 ] [d1b9ebf8c605] got first
state from fresh master [9nNJp0wHSi2sa44Cyk37Dw]
[2015-02-05 20:54:11,437][TRACE][discovery.ec2 ] [d1b9ebf8c605] updated
cluster join cluster to [1]
[2015-02-05 20:54:11,439][INFO ][cluster.service ] [d1b9ebf8c605]
detected_master
[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true},
added
{[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true},[bc7cbb4635f9][c_ch7LR1QAmYBQDUGbtTiQ][bc7cbb4635f9][inet[/172.16.255.149:9302]]{master=true},},
reason: zen-disco-receive(from master
[[d199693592ed][9nNJp0wHSi2sa44Cyk37Dw][d199693592ed][inet[/172.16.250.83:9302]]{master=true}])
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3a433856-2629-413f-a02f-0b9154302bb1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.