# Optimizing Kibana / Elastic for SIEM to stop Discover: Gateway Timeout errors **URL:** \ **Category:** Elasticsearch\ **Created:** [January 9, 2019, 10:05am UTC](https://discuss.elastic.co/t/optimizing-kibana-elastic-for-siem-to-stop-discover-gateway-timeout-errors/163501 "2019-01-09T10:05:43Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![CamTheMan](https://avatars.discourse-cdn.com/v4/letter/c/3ec8ea/32.png) [@CamTheMan](https://discuss.elastic.co/u/CamTheMan)\ **Post date:** [January 9, 2019, 10:05am UTC](https://discuss.elastic.co/t/optimizing-kibana-elastic-for-siem-to-stop-discover-gateway-timeout-errors/163501/1 "2019-01-09T10:05:43Z")
Hello, I have a 5 server cluster running each with: 24 CPU Cores 64 GB RAM (30 for elastic the rest for file caches) Lots of SSD disk (No spinners) We have: Nodes: **5** Indices: **455** Memory: **38.9 GB / 154.5 GB** Total Shards: **4329** Unassigned Shards: **0** Documents: **2,745,342,366** Data: **2.0 TB** The system is being using as a logging system SIEM etc, it's been running for about 5 months now, but when I run searches across an index-pattern Kibana will often error with: "Discover: Gateway Timeout errors" Now I know as the admin, that the search is running in the background, as usually if the user waits another 30+ seconds and reclicks 'search' in Kibana the results will be returned. Is there anyway to optimize the end-user experience for slow running searches - instead of getting the error: "Discover: Gateway Timeout errors" in Kibana? Thanks Cam ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 9, 2019, 10:28am UTC](https://discuss.elastic.co/t/optimizing-kibana-elastic-for-siem-to-stop-discover-gateway-timeout-errors/163501/2 "2019-01-09T10:28:24Z")
I would say that one of the problems you are facing is that you have far too many shards given the volume of data and size of cluster. Please read [this blog post on shards and sharing practices](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster) and then try to reduce this significantly. ---
**Author:** ![CamTheMan](https://avatars.discourse-cdn.com/v4/letter/c/3ec8ea/32.png) [@CamTheMan](https://discuss.elastic.co/u/CamTheMan)\ **Post date:** [January 9, 2019, 10:33am UTC](https://discuss.elastic.co/t/optimizing-kibana-elastic-for-siem-to-stop-discover-gateway-timeout-errors/163501/3 "2019-01-09T10:33:50Z")
Interesting. I have been reindexing our daily index patterns into monthly index patterns to consolidate shards. We did have ~ 16,000 shards about a month ago.... I'll keep working on reducing shard count... Thanks ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 9, 2019, 10:40am UTC](https://discuss.elastic.co/t/optimizing-kibana-elastic-for-siem-to-stop-discover-gateway-timeout-errors/163501/4 "2019-01-09T10:40:43Z")
2TB across 4329 shards is less than 500MB on average, which is very small. I would look to reduce the shard count by at least a factor of 10. As outlined in the blog post, an average shard size of a few tens of GB is quite reasonable and common. If you have a long retention period you may also want to have a look at [this webinar](https://www.elastic.co/webinars/optimizing-storage-efficiency-in-elasticsearch). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 6, 2019, 10:40am UTC](https://discuss.elastic.co/t/optimizing-kibana-elastic-for-siem-to-stop-discover-gateway-timeout-errors/163501/5 "2019-02-06T10:40:44Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.