ElasticSearch Cluster Settings Tuneup

(mohankumar) #1

Hi ,

I have maintain a ES Cluster with 10 Boxes each with 24 CPUS and 64 GB memory with separate master nodes.Per day we have written 4 Billion docs about 2 TB of Storage .

We have create a index with 30 shards.i have mentioned below the index settings.If any ware we have to tune the cluster settings.we have a huge bulk indexing and less queries.Still cluster not getting stable.

Any one will suggest the following settings.

In Bulk Writing environment , Do we have to change any other cluster settings ? Here we are using Spinner Disks.

Right now we are running in elasticsearch 1.5.2 Version

Thanks in the advance.

"settings" : {    
    "number_of_shards" : 30,
    "number_of_replicas": 0,
"refresh_interval" : -1,
    "index.translog.flush_threshold_size": "2147483648",
"index.translog.interval": "30s",



(Mark Walkom) #2

That's too many, you should reduce that to stop resource wastage.

(mohankumar) #3

Thanks Mark for your reply

But one more thing in a particular index we have to store 50 - 60 Billion Docs .So we have required minimum no of 30 shards.So you are saying we have to use multiple indexes or any other suggestions.

(Mark Walkom) #4

Oh right, then you're not going to be able to get around that anyway.

(system) #5