Showing high segments count

Elastic Stack Elasticsearch
1

I am using ES 2.4 and pushing the logs from logstash 2.4 with 20 workers (having 24 cores). I am seeing that
7911 elastics 20 0 37.4g 9.2g 390m S 190.2 29.4 1380:58 java

ES is using total memory(34 gb) of the server. After digging a lot I found that my ES segment count is too high.

My index details:
green open smsc_logs-2016.11.14 5 0 108885173 0 23.3gb 23.3gb

My segement details using _nodes/stats
"segments": { "count": 336, "memory_in_bytes": 182786916, "terms_memory_in_bytes": 162519572, "stored_fields_memory_in_bytes": 19560032, "term_vectors_memory_in_bytes": 0, "norms_memory_in_bytes": 22272, "doc_values_memory_in_bytes": 685040, "index_writer_memory_in_bytes": 6774424, "index_writer_max_memory_in_bytes": 851483029, "version_map_memory_in_bytes": 427825, "fixed_bit_set_memory_in_bytes": 0 },

From here I read that segments will tell you the number of Lucene segments this node currently serves. This can be an important number. Most indices should have around 50–150 segments, even if they are terabytes in size with billions of documents.

How to reduce this segment number??? I cant reduce the workers because I don't want to decrease my indexing speed. Is there any effective ways to reduce this segment counts?????

2

have you try force_merge? https://www.elastic.co/guide/en/elasticsearch/reference/2.4/indices-forcemerge.html

3

ES automatically handles segment merging, I wouldn't be too worried to be honest.

What is the actual problem you are trying to solve here?

4

https://www.elastic.co/guide/en/elasticsearch/guide/current/_monitoring_individual_nodes.html#_threadpool_section

segments will tell you the number of Lucene segments this node currently serves. This can be an important number. Most indices should have around 50–150 segments, even if they are terabytes in size with billions of documents.

In my index I have only 30GB data (16 crore docs) but I have 336 segments while indexing. And also os.memory.used_percent is 99. I want to reduce these stats (Segements & memory).

smsc_logs-2016.11.14 1 p 172.16.23.14 _12si 50274 23841810 0 4.4gb 20234283 true true 5.5.2 false smsc_logs-2016.11.14 0 p 172.16.23.14 _1148 48104 25835731 0 4.9gb 22151776 true true 5.5.2 false smsc_logs-2016.11.14 0 p 172.16.23.14 _17ka 56458 208474 0 39.7mb 213748 true true 5.5.2 true smsc_logs-2016.11.14 0 p 172.16.23.14 _1714 55768 153416 0 26.4mb 144906 true true 5.5.2 true

Please explain How to make this segments low & the os.memory get released from ES?????

5

The OS handles release of off heap memory, how to handle that is outside the scope of these forums though.

6

So is it ok to have these counts???

"segments": { "count": 201,

"mem": { "total_in_bytes": 33694126080, "free_in_bytes": 295096320, "used_in_bytes": 33399029760, "free_percent": 1, "used_percent": 99

Are you saying the problem is nothing with these stats(Segment count & memory)????

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.