# Very large segments - Do I need to worry? **URL:** \ **Category:** Elasticsearch\ **Created:** [January 8, 2019, 8:37am UTC](https://discuss.elastic.co/t/very-large-segments-do-i-need-to-worry/163317 "2019-01-08T08:37:32Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Martin\_Becker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martin_becker/32/22885_2.png) [@Martin\_Becker](https://discuss.elastic.co/u/Martin_Becker)\ **Post date:** [January 8, 2019, 8:37am UTC](https://discuss.elastic.co/t/very-large-segments-do-i-need-to-worry/163317/1 "2019-01-08T08:37:32Z")
We run many dashboards based on our data. As requests were getting slow a year ago we came up with the idea of merging segments in our indices and had a performance boost of around 20%. After visiting an Elastic training program here in Berlin I learned about the downsides of segment reducing. So I stopped the automatic segment reducing since it could lead to unwanted effects. I started to investigate the current situation. Here is what I found: Request ``` GET /_cat/segments?v&index=analytics-migrations-201812&s=size:desc ``` Response ``` index shard prirep ip segment generation docs.count docs.deleted size size.memory committed searchable version compound analytics-migrations-201812 0 p 172.24.24.191 _np3k 1105616 31439133 10515132 5gb 2185544 true true 6.6.1 false analytics-migrations-201812 0 r 172.24.24.192 _x24u 1542414 28059413 8649610 4.4gb 1930234 true true 6.6.1 false analytics-migrations-201812 0 r 172.24.24.192 _k36p 937249 22719776 8502457 3.7gb 1648627 true true 6.6.1 false analytics-migrations-201812 0 p 172.24.24.191 _x7p7 1549627 20575200 8714796 3.5gb 1549248 true true 6.6.1 false analytics-migrations-201812 0 r 172.24.24.192 _14slg 1903300 14213376 3519637 2.1gb 969675 true true 6.6.1 false analytics-migrations-201812 0 p 172.24.24.191 _14f9e 1886018 12796404 3549667 2gb 895393 true true 6.6.1 false analytics-migrations-201812 0 r 172.24.24.192 _1bekx 2211729 7316133 5329274 1.5gb 698780 true true 6.6.1 false analytics-migrations-201812 0 p 172.24.24.191 _19yah 2143961 3840324 4702690 1gb 499394 true true 6.6.1 false analytics-migrations-201812 0 p 172.24.24.191 _1bekj 2211715 4121995 2634361 855.9mb 407861 true true 6.6.1 true analytics-migrations-201812 0 p 172.24.24.191 _1dsm1 2323225 4185331 39887 528.4mb 279747 true true 6.6.1 true analytics-migrations-201812 0 r 172.24.24.192 _1dg86 2307174 2341967 657786 374.3mb 215218 true true 6.6.1 true analytics-migrations-201812 0 p 172.24.24.191 _1cypu 2284482 1523517 226437 217.4mb 124784 true true 6.6.1 true ... ``` The index "analytics-migrations-201812" consists of around 85 segments. As one can see there are some segments which are huge holding many deleted docs. Some of them are very large but with compound=false. I wonder about the compound flag. After looking up the meaning of it I am not sure what it actually tells. Only the huge ones are tagged with "compound=false". Is there a potential problem? Do I need to take action? What could be done if so? Thank you. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 8, 2019, 8:46am UTC](https://discuss.elastic.co/t/very-large-segments-do-i-need-to-worry/163317/2 "2019-01-08T08:46:55Z")
It looks like you are using time-based indices, which usually means retention is managed by simply dropping the full index. How come you are deleting documents from them? ---
**Author:** ![Martin\_Becker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martin_becker/32/22885_2.png) [@Martin\_Becker](https://discuss.elastic.co/u/Martin_Becker)\ **Post date:** [January 8, 2019, 8:59am UTC](https://discuss.elastic.co/t/very-large-segments-do-i-need-to-worry/163317/3 "2019-01-08T08:59:52Z")
The "analytics-migration-YYYYMM" indices hold documents (events) which can be updated based on new events which can have a relation to each other. That's how all the deleted documents get generated. We know that's not ideal but we need to read the relations from one document to the other if there is one. As far as I know the root cause of the mistake was segment reducing of read-write-indices. We cannot simply delete the index because our dashboards will query for the data. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 5, 2019, 9:00am UTC](https://discuss.elastic.co/t/very-large-segments-do-i-need-to-worry/163317/4 "2019-02-05T09:00:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.