No document found after restart one node


(immanens) #1

Hi,
For unknown reasons, one of node shows 0 document and 0 storage in elastichq plugin.

root@esnode-2:~# curl 'localhost:9200/_cat/indices?v'
health status index pri rep docs.count docs.deleted store.size pri.store.size
yellow open my-prod 5 1 18434228 4836235 584.7gb 584.7gb

I got these logs:
[2017-04-26 10:07:34,272][TRACE][discovery.zen.ping.unicast] [esnode2] [1] disconnecting from [#zen_unicast_1#][esnode-2][inet[/10.0.0.134:9300]]
[2017-04-26 10:07:34,276][TRACE][discovery.zen ] [esnode2] full ping responses:
--> ping_response{node [[esnode1][3nl5AzuAQQ2BPHnqZrTmwg][esnode-1][inet[/10.0.0.134:9300]]], id[60], master [[esnode1][3nl5AzuAQQ2BPHnqZrTmwg][esnode-1][inet[/10.0.0.134:9300]]], hasJoinedOnce [true], cluster_name[escluster1]}
[2017-04-26 10:07:34,276][DEBUG][discovery.zen ] [esnode2] filtered ping responses: (filter_client[true], filter_data[false])
--> ping_response{node [[esnode1][3nl5AzuAQQ2BPHnqZrTmwg][esnode-1][inet[/10.0.0.134:9300]]], id[60], master [[esnode1][3nl5AzuAQQ2BPHnqZrTmwg][esnode-1][inet[/10.0.0.134:9300]]], hasJoinedOnce [true], cluster_name[escluster1]}
[2017-04-26 10:07:34,284][TRACE][discovery.zen ] [esnode2] joining master [esnode1][3nl5AzuAQQ2BPHnqZrTmwg][esnode-1][inet[/10.0.0.134:9300]]
[2017-04-26 10:07:34,302][DEBUG][discovery.zen.publish ] [esnode2] received cluster state version 23002
[2017-04-26 10:07:34,303][DEBUG][discovery.zen.fd ] [esnode2] [master] restarting fault detection against master [[esnode1][3nl5AzuAQQ2BPHnqZrTmwg][esnode-1][inet[/10.0.0.134:9300]]], reason [new cluster state received and we are monitoring the wrong master [null]]
[2017-04-26 10:07:34,304][DEBUG][discovery.zen ] [esnode2] got first state from fresh master [3nl5AzuAQQ2BPHnqZrTmwg]
[2017-04-26 10:07:34,304][TRACE][discovery.zen ] [esnode2] updated cluster join cluster to [1]

this line is bizarre : new cluster state received and we are monitoring the wrong master

and here's esnode config:

cluster.name: escluster1
node.name: "esnode1"
path.data: /var/data/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.mlockall: true
gateway.recover_after_nodes: 1
gateway.recover_after_time: 5m
gateway.expected_nodes: 2
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["10.0.0.13:9300"]
threadpool.bulk.type: fixed
threadpool.bulk.queue_size: 1200

cluster.name: escluster1
node.name: "esnode2"
path.data: /var/data/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.mlockall: true
gateway.recover_after_nodes: 1
gateway.recover_after_time: 5m
gateway.expected_nodes: 2
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["10.0.0.134:9300"]
threadpool.bulk.type: fixed
threadpool.bulk.queue_size: 1200

tcpdump shows that zen ping is passing between both nodes. all physical files are present on the second node : esnode-2, but it seems ignore it :slight_smile:
Indices

Documents: 0
Documents Deleted: 0
Store Size: 0.0
Index Req Total: 0
Delete Req Total: 0
Get Req Total: 0
Get(Exists) Total: 0
Get(Missing) Total: 0
Query Total: 0

curl 'localhost:9200/_cluster/health?level=indices&pretty'
{
"cluster_name" : "escluster1",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 5,
"active_shards" : 5,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 5,
"number_of_pending_tasks" : 0,
"indices" : {
"retronews-prod" : {
"status" : "yellow",
"number_of_shards" : 5,
"number_of_replicas" : 1,
"active_primary_shards" : 5,
"active_shards" : 5,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 5
}
}
}

I restarted it, and node 2 discovered 2 of 5 shades , slooooooooowwwwly , so I added some memory and reboot the server, after 15hours, it shows 0 documents and "unassigned_shards" : 5

thanks for your helps:
regards


(Mark Walkom) #2

What version are you on?


(immanens) #3

ii elasticsearch 1.5.2 all


(immanens) #4

i got this
curl -s 'localhost:9200/_cat/allocation?v'
shards disk.used disk.avail disk.total disk.percent host ip node
0 5.1tb 755.1gb 5.9tb 87 esnode-2 10.0.0.13 esnode2
5 5.1tb 755.1gb 5.9tb 87 esnode-1 74.247.236.178 esnode1
5 UNASSIGNED

esnode1 do not have right IP. this is an old IP. I don't known why this IP is here, and how to change it.


(immanens) #5

By the way, 87% used disk space, can it be a problem?


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.