HW recommendations and best practices for a big index rate(3TB/day)

Elastic Stack Elasticsearch
1

Hi All,

Elasticsearch site provides a fairly minimal HW recommendations regarding on what to use to best utilize ES ("use SSDs), and provides no estimates on what performance should be expected according to the defined HW and i hoped that i could get recommendations on what we should use on my current setup.

currently we are generating about 2.5-3TB of data/day (about 2B documents per day) and I save it for a 7 days.
my HW specs are as follows:

3 elastic nodes of the following -
48 cores
64 GB RAM (30.5 for java heap, rest for OS as per elastic recommendations).
10K spinning disks (no raid) .

4 client nodes streaming data using bulk API.

ES is 1.7.2

normal indexing work can keep up, but when running queries on the data using kibana after some time i'm starting to see a lot of GC WARN messages (GC messages every 15 minutes or so), and when i start seeing '[FIELDDATA] Data too large' errors the cluster hangs and stops responding on GC goes into overdrive.

also i am seeing alot of 'now throttling indexing: numMergesInFlight=6, maxNumMerges=5' messages, and in HQ Field Evictions number keeps growing.

first of all I want to know if its even possible for my current setup to handle such traffic?

I know that the disks are a big bottleneck here but i want to know is there something to be done except upgrading storage? and is there a way to test how much indexing rate my current setup is able to handle?

If you need additional information please let me know.

Thanks!

2

This is not "normal indexing work can keep up", this is the opposite. You need to add more nodes to handle indexing and searching simultaneously.

Also you should examine your ingest process, maybe you write new data into the same indices where queries are executed with filters/aggregations. This has a significant impact on the required resources and the overall performance. It is much better to decouple read from write, i.e. first, bulk index, maybe tune segment count, then switch the index to get included into the search. The effectiveness of such a procedure depends on how fresh the data must be that has to be searched, of course.

3

Thanks for the reply,

what i mean by saying "normal indexing work can keep up", is that when no searching is done ad hoc on the cluster , the cluster can keep up with the data, shard is allocated properly and cluster stays in green status.
when searches start, that's when the cluster struggles, i get many GC and index throttle messages in the logs.

there is something i dont understand here, how would the addition of more nodes help with the indexing? the write load would split to the additional nodes?

we do write new data to the same indices where queries are being executed. we read from logs via seperate streams (1 stream per log type) and create a daily index to each log type for 1 week, and delete indexes older then 1 week.

could you please advice on how to technically "decouple reads from writes" because i didnt see any option on how to limit access to actively written indexes.
how do you configure such a thing?

Thanks.

4

Also maybe this will help a little bit?

5

@voipoclay thanks for the link, our refresh_interval is already at 30s.
i wonder if we increase it even further from 1-5 minutes, it will help (our usecases can handle a few minutes data delays).

6

Writing into indices that are just being searched is a delicate operation.

Depending on the kind of search, ES will suddenly demand huge memory resources, by loading segments from disk, and by allocating memory for filter/aggregations. These resources are only valid for the time when the next index segment merging is done and creates new segments. Then old segments are dropped and search results are invalidated. From then on, queries must rebuild the filters/aggregations again and cannot reuse the dropped segments. Depending on the frequency of such events, nodes can show real high resource usage spikes for some seconds.

If you really must adhere to this operation style, you can remedy the situation a bit, by adding nodes to spread the resource usage spikes. The idea is to keep the volume of the moving data low per node, so the refresh becomes not noticeable.

My ETL platform that I implemented writes always into a new index. I can also control on which nodes the index shards are allocated (but, search load is not high, so I don't care about search-only dedication) When index is ready, I switch the search application over to the new index. This is done by removing the index alias from the old index and putting it to the new one, which is atomic (no downtime).