Elastic Server automatically shutdown

Elastic Stack Elasticsearch
1

I have three elastic servers all masters having total 32 GB ram each,16 GB allocated to elastic servers.
From last few days any one of three servers randomly automatically goes down and the java.exe is killed automatically.Even no error is logged in elastic search logs.

Am using virtual environment with network file system to store data and logs
Elastic search version is 1.7.0
Indices count:around 300
Total documents in whole elastic search : around 60 million

Below is my config

cluster.name: Cluster1 cluster.routing.allocation.disk.threshold_enabled: false script.disable_dynamic: false node.name: "Master1" node.master: true node.data: true index.query.bool.max_clause_count: 50100 indices.fielddata.cache.size: 25% indices.fielddata.cache.expire: 5m action.disable_delete_all_indices: true indices.cluster.send_refresh_mapping: false index.cache.field.type: soft path.data: \nas5\Elasticsearch\Data path.logs: \nas5\Elasticsearch\Logs\Master1 bootstrap.mlockall: true http.max_content_length: 999mb indices.recovery.max_bytes_per_sec: 100mb indices.recovery.concurrent_streams: 5

above config goes same for all three servers Master1,Master2,Master3.

2

Then something external is stopping the service, if ES is told to stop then it will log something.

3

Thanks for your reply.
Yes am checking that too,if something external is stopping the service.
Still not figured out what is causing the stop.
Is there anything you could help me out in some settings in elasitc server

4

What do you see in the logs when it went down? That might give you some clue.

5

logs have regular logs about indexing,GC collection etc and nothing related to error

6

You have a number of settings that could drive up memory usage, e.g. "index.query.bool.max_clause_count: 50100" and "http.max_content_length: 999mb". You also seem to have a quite large number of indices considering the number of documents in the cluster. Could you please tell us a bit more about the use case and the rationale behind these settings?

7

I am storing content of the document which i use for MLT query.The reason behind setting index.query.bool.max_clause_count: 50100 is because sometimes need to make huge term query which defaults to 1024 clause count.The reason behind setting http.max_content_length: 999mb is because i need to index content of documents and the default content length is 100mb which was causing error due to very large content documents.

8

Any updates?