Limiting Segment Memory Consumption

Elastic Stack Elasticsearch
1

We have been running under constant memory pressure on our ES nodes. Upon analyzing it appears that segment is consuming more than 50% of the available heap. Is there a way to configure the amount of memory that can be used for caching segment data?

We though about reducing shards, but then that would lead to very large shards, our index is about 1TB. Is there a better practice / strategy to use in such cases?

We are running 1.7.

2

How many shards do you have?

3

Currently - 24 - its coming down to about 40gb per shard. I understand it not ideal but data grew faster than expected.

4

40GB is fine.

What sort of data are you dealing with?

5

Data is quite heterogeneous - but largely its e-commerce / retail orders including customer requests with fields such as array of comments, phone number, email id, dates etc. We also have international customers and so some non-english characters (largely east european) in there. I checked the .tim files on the disk and they added up to 40gb.

For shard size - i thought 40 gb was crossing the "reasonable" limit as suggested in a few online posts. Is there such a limit for a shard where query latency and / or shard replication / relocation is severely hampered?

6

If I remember correctly, the somewhat ideal shard size is under 50GB from a presentation by @warkolm

7

And you're putting this all into a single index? Sounds like you need to break it up a bit more, it only time based by order time.

That's wrong, use the APIs.

8

I did use the api but couldn't find if the term dictionary size is exposed by the api. The memory consumed by segments was from the api.

We are evaluating whether time based could work in our functional scenario.

On the original question though - is there a way to limit the amount of heap used by segments?

9

Can you share where you are seeing the segments using all the memory?
Ie the output of the API call?

10

Here is the snippet -

segments" : {
"count" : 969,
"memory_in_bytes" : 46613585150,
"index_writer_memory_in_bytes" : 5838763,
"index_writer_max_memory_in_bytes" : 8252512238,
"version_map_memory_in_bytes" : 28576,
"fixed_bit_set_memory_in_bytes" : 78476616
}

Appreciate your help!