Hi
We have deployed elasticsearch 2.3.5 across 22 nodes. We have opened firewall for port 9200 and 9300 from our client nodes to the ES cluster.
We noticed that the curl commands for PUT works perfectly fine from the client nodes, while the GET for search fails with error 56. This is confusing because we are able to query the index health and shard from the same client nodes successfully. All the queries works from the ES nodes however.
The indexes are created at runtime based on the current date.
Below is a sample..
curl -XPUT 'http://searchserv1.lvs.paypal.com:9200/ss-20160831/raw_data/1' -d '{"alpha":"foo2","beta":"bar2"}'
{"_index":"fpti-20160831","_type":"raw_data","_id":"1","_version":3,"_shards":{"total":2,"successful":2,"failed":0},"created":false}
curl -XGET 'http://searchserv1.lvs.paypal.com:9200/ss-20160831/raw_data/_search?q=alpha:foo2'
curl: (56) Failure when receiving data from the peer
curl -XGET 'http://searchserv1.lvs.paypal.com:9200/_cluster/health?level=indices&pretty'
{
"cluster_name" : "mysearch",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 19,
"number_of_data_nodes" : 19,
"active_primary_shards" : 121,
"active_shards" : 242,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0,
"indices" : {
"ss-20160831" : {
"status" : "green",
"number_of_shards" : 40,
"number_of_replicas" : 1,
"active_primary_shards" : 40,
"active_shards" : 80,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"ss-20160830" : {
"status" : "green",
"number_of_shards" : 40,
"number_of_replicas" : 1,
"active_primary_shards" : 40,
"active_shards" : 80,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
".kibana" : {
"status" : "green",
"number_of_shards" : 1,
"number_of_replicas" : 1,
"active_primary_shards" : 1,
"active_shards" : 2,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
},
"ss-20160829" : {
"status" : "green",
"number_of_shards" : 40,
"number_of_replicas" : 1,
"active_primary_shards" : 40,
"active_shards" : 80,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}
}
}
Any suggestions/recommendations would be highly appreciated.
Thanks in advance