ElasticSearch Cluster Settings Tuneup

Hi ,

I have maintain a ES Cluster with 10 Boxes each with 24 CPUS and 64 GB memory with separate master nodes.Per day we have written 4 Billion docs about 2 TB of Storage .

We have create a index with 30 shards.i have mentioned below the index settings.If any ware we have to tune the cluster settings.we have a huge bulk indexing and less queries.Still cluster not getting stable.

Any one will suggest the following settings.

In Bulk Writing environment , Do we have to change any other cluster settings ? Here we are using Spinner Disks.

Right now we are running in elasticsearch 1.5.2 Version

Thanks in the advance.

"settings" : {    
    "number_of_shards" : 30,
    "number_of_replicas": 0,
"refresh_interval" : -1,
    "index.translog.flush_threshold_size": "2147483648",
"index.translog.interval": "30s",



That's too many, you should reduce that to stop resource wastage.

Thanks Mark for your reply

But one more thing in a particular index we have to store 50 - 60 Billion Docs .So we have required minimum no of 30 shards.So you are saying we have to use multiple indexes or any other suggestions.

Oh right, then you're not going to be able to get around that anyway.