We have a quite unique set up with ES1.7 running on Windows Server 2008 R2. Also, we are not running ES standalone but instead embedding the Elasticsearch jar files in our own web app running in tomcat containers and accessing ES via the java API.
We initially had 3 nodes and experienced node drops (one of the nodes not reachable and then a few minutes later connection re-established) frequently. During the node drops the following messages can be seen in our logs:
· [DEBUG] 2016-06-13 18:12:10,061 org.elasticsearch.indices.memory - [prmwntu9] marking shard [end_device_events_20160227][1] as inactive (inactive_time[5m]) indexing wise, setting size to [500kb]
· 2016-06-13 18:12:10,061 org.elasticsearch.index.shard - [prmwntu9] [end_device_events_20160227][1] updating index_buffer_size from [98.7mb] to (inactive) [500kb]
· [DEBUG] 2016-06-13 18:12:10,061 org.elasticsearch.indices.memory - [prmwntu9] recalculating shard indexing buffer (reason=[[ADDED, BECAME_INACTIVE]]), total is [2.9gb] with [13] active shards, each shard set to indexing=[235.4mb], translog=[64kb]
· [DEBUG] 2016-06-13 18:12:49,732 org.elasticsearch.index.cache.filter.weighted - [prmwntu9] [meter_readings_20160323] full cache clear, reason [close]
· 2016-06-13 18:12:49,732 org.elasticsearch.index.cache.fixedbitset - [prmwntu9] [meter_readings_20160323] clearing all bitsets because [close]
below is the network stats after a fresh cluster restart:
"network" : {
"tcp" : {
"active_opens" : 24932381,
"passive_opens" : 53804945,
"curr_estab" : 726,
"in_segs" : 918135524,
"out_segs" : 738127487,
"retrans_segs" : 1089623,
"estab_resets" : 24684121,
"attempt_fails" : 104055, <---- not sure if this is a problem
"in_errs" : 0,
"out_rsts" : 29521839
We attempted to add 2 more nodes to make it a 5 nodes system and after that the drop frequency increased multiple-folds (from 1 or 2 a day to over 30 a day). This is happening with or without load and can happen at any time.
Searching the forum did not turn out anything. So any suggestions from anybody is greatly appreciated.