Hi,
For unknown reasons, one of node shows 0 document and 0 storage in elastichq plugin.
root@esnode-2:~# curl 'localhost:9200/_cat/indices?v'
health status index pri rep docs.count docs.deleted store.size pri.store.size
yellow open my-prod 5 1 18434228 4836235 584.7gb 584.7gb
I got these logs:
[2017-04-26 10:07:34,272][TRACE][discovery.zen.ping.unicast] [esnode2] [1] disconnecting from [#zen_unicast_1#][esnode-2][inet[/10.0.0.134:9300]]
[2017-04-26 10:07:34,276][TRACE][discovery.zen ] [esnode2] full ping responses:
--> ping_response{node [[esnode1][3nl5AzuAQQ2BPHnqZrTmwg][esnode-1][inet[/10.0.0.134:9300]]], id[60], master [[esnode1][3nl5AzuAQQ2BPHnqZrTmwg][esnode-1][inet[/10.0.0.134:9300]]], hasJoinedOnce [true], cluster_name[escluster1]}
[2017-04-26 10:07:34,276][DEBUG][discovery.zen ] [esnode2] filtered ping responses: (filter_client[true], filter_data[false])
--> ping_response{node [[esnode1][3nl5AzuAQQ2BPHnqZrTmwg][esnode-1][inet[/10.0.0.134:9300]]], id[60], master [[esnode1][3nl5AzuAQQ2BPHnqZrTmwg][esnode-1][inet[/10.0.0.134:9300]]], hasJoinedOnce [true], cluster_name[escluster1]}
[2017-04-26 10:07:34,284][TRACE][discovery.zen ] [esnode2] joining master [esnode1][3nl5AzuAQQ2BPHnqZrTmwg][esnode-1][inet[/10.0.0.134:9300]]
[2017-04-26 10:07:34,302][DEBUG][discovery.zen.publish ] [esnode2] received cluster state version 23002
[2017-04-26 10:07:34,303][DEBUG][discovery.zen.fd ] [esnode2] [master] restarting fault detection against master [[esnode1][3nl5AzuAQQ2BPHnqZrTmwg][esnode-1][inet[/10.0.0.134:9300]]], reason [new cluster state received and we are monitoring the wrong master [null]]
[2017-04-26 10:07:34,304][DEBUG][discovery.zen ] [esnode2] got first state from fresh master [3nl5AzuAQQ2BPHnqZrTmwg]
[2017-04-26 10:07:34,304][TRACE][discovery.zen ] [esnode2] updated cluster join cluster to [1]
this line is bizarre : new cluster state received and we are monitoring the wrong master
and here's esnode config:
cluster.name: escluster1
node.name: "esnode1"
path.data: /var/data/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.mlockall: true
gateway.recover_after_nodes: 1
gateway.recover_after_time: 5m
gateway.expected_nodes: 2
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["10.0.0.13:9300"]
threadpool.bulk.type: fixed
threadpool.bulk.queue_size: 1200
cluster.name: escluster1
node.name: "esnode2"
path.data: /var/data/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.mlockall: true
gateway.recover_after_nodes: 1
gateway.recover_after_time: 5m
gateway.expected_nodes: 2
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["10.0.0.134:9300"]
threadpool.bulk.type: fixed
threadpool.bulk.queue_size: 1200
tcpdump shows that zen ping is passing between both nodes. all physical files are present on the second node : esnode-2, but it seems ignore it 
Indices
Documents: 0
Documents Deleted: 0
Store Size: 0.0
Index Req Total: 0
Delete Req Total: 0
Get Req Total: 0
Get(Exists) Total: 0
Get(Missing) Total: 0
Query Total: 0
curl 'localhost:9200/_cluster/health?level=indices&pretty'
{
"cluster_name" : "escluster1",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 5,
"active_shards" : 5,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 5,
"number_of_pending_tasks" : 0,
"indices" : {
"retronews-prod" : {
"status" : "yellow",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 5,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 5
}
}
}
I restarted it, and node 2 discovered 2 of 5 shades , slooooooooowwwwly , so I added some memory and reboot the server, after 15hours, it shows 0 documents and "unassigned_shards" : 5
thanks for your helps:
regards