ElasticSearch Cluster Settings Tuneup

Elastic Stack Elasticsearch
1

Hi ,

I have maintain a ES Cluster with 10 Boxes each with 24 CPUS and 64 GB memory with separate master nodes.Per day we have written 4 Billion docs about 2 TB of Storage .

We have create a index with 30 shards.i have mentioned below the index settings.If any ware we have to tune the cluster settings.we have a huge bulk indexing and less queries.Still cluster not getting stable.

Any one will suggest the following settings.

In Bulk Writing environment , Do we have to change any other cluster settings ? Here we are using Spinner Disks.

Right now we are running in elasticsearch 1.5.2 Version

Thanks in the advance.

"settings" : {    
    "number_of_shards" : 30,
    "number_of_replicas": 0,
    "index.store.compress.stored":"true",
    "index.store.compress.tv":"true",
     "compress.default.type":"snappy",
"index.query.default_field":"oplog",
"refresh_interval" : -1,
    "index.merge.policy.max_merge_at_once":10,
    "index.translog.flush_threshold_size": "2147483648",
"index.translog.interval": "30s",
    "index.store.type":"mmapfs"

}

Thanks
Moni

2

That's too many, you should reduce that to stop resource wastage.

3

Thanks Mark for your reply

But one more thing in a particular index we have to store 50 - 60 Billion Docs .So we have required minimum no of 30 shards.So you are saying we have to use multiple indexes or any other suggestions.

4

Oh right, then you're not going to be able to get around that anyway.