# Performance Problems **URL:** \ **Category:** Elasticsearch\ **Tags:** user-experience\ **Created:** [November 27, 2023, 10:51am UTC](https://discuss.elastic.co/t/performance-problems/348035 "2023-11-27T10:51:16Z")\ **Posts on this page:** 9\ **Page:** 2
**Author:** ![fgonzalez](https://avatars.discourse-cdn.com/v4/letter/f/71e660/32.png) [@fgonzalez](https://discuss.elastic.co/u/fgonzalez)\ **Post date:** [December 7, 2023, 3:50pm UTC](https://discuss.elastic.co/t/performance-problems/348035/21 "2023-12-07T15:50:55Z")
Good afternoon @Christian_Dahlqvist I am going to escalate your recommendations and we will try to implement them as soon as possible, I just wanted to clarify a few things in case this implies changing any parameter of what was discussed. The 4 TB of data is for each HOT node of the 6 that you have, that is, 24TB in the HOT part between the information + the replica. I will tell you more details and updated information as soon as I have it, thank you for your help! **Sorry to emphasize this again, but doesn't separating the 700TB of total data into more than one indexset have a significant impact on performance? At the end of the day it is like having a SQL with only one monstrous table where all the data is entered in a nutshell.** thanks greetings! ---
**Author:** ![fgonzalez](https://avatars.discourse-cdn.com/v4/letter/f/71e660/32.png) [@fgonzalez](https://discuss.elastic.co/u/fgonzalez)\ **Post date:** [December 21, 2023, 2:53pm UTC](https://discuss.elastic.co/t/performance-problems/348035/22 "2023-12-21T14:53:25Z")
Good afternoon, We have planned to change the size of the shards for the week of January 2nd, I will tell you the news. Thank you and merry Christmas! ---
**Author:** ![fgonzalez](https://avatars.discourse-cdn.com/v4/letter/f/71e660/32.png) [@fgonzalez](https://discuss.elastic.co/u/fgonzalez)\ **Post date:** [January 9, 2024, 7:44am UTC](https://discuss.elastic.co/t/performance-problems/348035/23 "2024-01-09T07:44:34Z")
Good morning, We will finally implement the changes next week, keeping you informed. Thanks greetings! ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 9, 2024, 8:23am UTC](https://discuss.elastic.co/t/performance-problems/348035/24 "2024-01-09T08:23:32Z")
> [@fgonzalez](#): > > Sorry to emphasize this again, but doesn't separating the 700TB of total data into more than one indexset have a significant impact on performance? At the end of the day it is like having a SQL with only one monstrous table where all the data is entered in a nutshell. I do not understand this comment/question. Elasticsearch works very differently compared to a SQL database. ---
**Author:** ![fgonzalez](https://avatars.discourse-cdn.com/v4/letter/f/71e660/32.png) [@fgonzalez](https://discuss.elastic.co/u/fgonzalez)\ **Post date:** [January 29, 2024, 1:14pm UTC](https://discuss.elastic.co/t/performance-problems/348035/25 "2024-01-29T13:14:32Z")
Good afternoon, I know it is not comparable, but I am not sure if it is the most optimal to put everything in the same indexset, because without going any further, sometimes we get warnings about reaching the limit of indexed fields of 1000, therefore if it is divided The information by affinity in different indexsets should a priori be more optimal, right? Regarding the change in the size of the shards, in principle we should have done it but it has become complicated and we have not yet been able to carry it out, I will keep you informed. Thanks greetings! ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 29, 2024, 1:30pm UTC](https://discuss.elastic.co/t/performance-problems/348035/26 "2024-01-29T13:30:16Z")
Best practice in general is to group data with similar mappings into index sets, and part of this is to avoid mapping explosion with respect to the number of fields. You do not want to go too granular though as you will end up with lots of very small indices and shards, which is very inefficient and hurts performance. It seems like Graylog only creates a single index set, which may indeed not be optimal. That is however something you would need to address with them. ---
**Author:** ![fgonzalez](https://avatars.discourse-cdn.com/v4/letter/f/71e660/32.png) [@fgonzalez](https://discuss.elastic.co/u/fgonzalez)\ **Post date:** [January 29, 2024, 1:32pm UTC](https://discuss.elastic.co/t/performance-problems/348035/27 "2024-01-29T13:32:22Z")
> [@Christian\_Dahlqvist](#): > > Best practice in general is to group data with similar mappings into index sets, and part of this is to avoid mapping explosion with respect to the number of fields. You do not want to go too granular though as you will end up with lots of very small indices and shards, which is very inefficient and hurts performance. > > It seems like Graylog only creates a single index set, which may indeed not be optimal. That is however something you would need to address with them. No, we can really have as many indexsets as we want, Graylog is not a problem for this. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 29, 2024, 1:34pm UTC](https://discuss.elastic.co/t/performance-problems/348035/28 "2024-01-29T13:34:47Z")
Then I would recommend creating multiple index sets based on how similar the mappings for different types of data are. As retention management is done at the index level, another aspect to consider is to also group data that have the same retention period together. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 26, 2024, 1:34pm UTC](https://discuss.elastic.co/t/performance-problems/348035/29 "2024-02-26T13:34:54Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/performance-problems/348035.md?page=1)