Slow Index create

Elastic Stack Elasticsearch
1

Hi,

We are struggling with ES performance issue, we have daily indexes and minute or two after midnight our cluster is unavailable. Logstash indexer reports time out, after around 10 min everything back to normal. we try few different strategies, create indexer with 0 replica and replica during the day. In pick we creates 150 indexes. After investigation we discovered index create time is around 8 sec, even after high pick still it's 8sec. We believe it's root cause.
My questions:

  1. How to speed up index creation ?

  2. Does it make sense have dedicated servers and VMs ? Does cluster speed depends on slowest node ?

Our cluster: 3 masters, 3 hot nodes (dedicated bare-metal serves) 6 cold nodes VMs, 1 ingest node.
We have 20k shards, 15k on cold nodes.
"version" : {
"number" : "6.4.2",
"build_flavor" : "default",
"build_type" : "rpm",
"build_hash" : "04711c2",
"build_date" : "2018-09-26T13:34:09.098244Z",
"build_snapshot" : false,
"lucene_version" : "7.4.0",

Thanks for any suggestions.

2

You can have a look at Rollover API or write a custom job to pre-create indices in advance (the idea is to spread mass index creation over a longer period of time).

HTH,
Lukáš

3

You have far too many shards, and this can indeed result in performance issues. Here is an article with some more detailed guidance:

4

Thanks for reply,

How many shards I can have on data nodes ? Psychical servers 256G RAM, 72CPU, fast disks.

I'm a little bit confused, in many places recommended numbers shards is 1K per node, but what if node is really fast server ?

What is better way to scale ElasticSearch vertical or horizontal ? May be it doesn't make sense pay for really fast servers, better invest in many small instances. I'm really curios about your opinion.

Thanks.

5

The recommendation is significantly less than 1000 shards per node. The guideline in the article I shared is 20 shards per GB of heap, and you should have less than 32GB of heap, so that puts the recommended limit at less than 640 shards. It also recommends a target shard size of 20-40GB, which adds up to 12-25TB per node. If you have more data than that then either target larger shards or else scale out horizontally.

1 Like
6

@DavidTurner thanks for explanation.

Please correct me if I'm wrong but I've feeling it's doesn't make sense buy expensive servers ( 256G RAM, 72CPU ) because of ES limit, heap should no more 32G. Seems optimal servers is around 64G RAM and scale out horizontally instead invest in expensive fast bare-metal server.

Right ?

7

The optimal hardware for your cluster very much depends on your needs and you can only really determine the best setup with careful benchmarking of a realistic workload. There are definitely workloads that benefit from 256GB or more of RAM. Elasticsearch will use any spare RAM for the filesystem cache, which can be many times larger than the JVM heap.

8

@DavidTurner and @Lukas_Vlcek1 thank you a lot ! I'm going back to my cluster, seems I need review our assumptions.
You can close this it.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.