# Kibana SIEM app performance **URL:** \ **Category:** SIEM\ **Created:** [May 26, 2020, 2:52pm UTC](https://discuss.elastic.co/t/kibana-siem-app-performance/234358 "2020-05-26T14:52:32Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![j91321](https://avatars.discourse-cdn.com/v4/letter/j/7ea924/32.png) [@j91321](https://discuss.elastic.co/u/j91321)\ **Post date:** [May 26, 2020, 2:52pm UTC](https://discuss.elastic.co/t/kibana-siem-app-performance/234358/1 "2020-05-26T14:52:32Z")
Hello, we are experiencing some performance issues in Kibana SIEM app especially loading the Detections tab. Making it very annoying to use. The queries report times ~8ms (Signal count) ~33ms (Inspect signals). But the detections tab takes around 30s to load and is very unresponsive. Clicking inspect on visualization makes the windows 5s unresponsive before anything happens. Also this seems to be worse in custom space than in default space. At first I was thinking detection engine is causing this since we use a 400+ rules. So I have increased workers value to 100. Same result. So I have disabled all rules and nothing really changed. I have noticed in Kibana logs one POST request takes long (I have truncated the log): ```auto "message":"POST /api/detection_engine/signals/search 200 112ms - 9.0B" "message":"POST /api/siem/graphql 200 741ms - 9.0B" "message":"POST /api/siem/graphql 200 122ms - 9.0B" "message":"POST /api/ui_metric/report 200 607ms - 9.0B" "message":"POST /api/ui_metric/report 200 640ms - 9.0B" "message":"POST /api/siem/graphql 200 1364ms - 9.0B" "message":"POST /api/siem/graphql 200 3091ms - 9.0B" "message":"POST /api/ui_metric/report 200 533ms - 9.0B" "message":"POST /api/siem/graphql 200 2366ms - 9.0B" ``` Sometimes calls to `/api/siem/graphql` take around ~2000-3000ms and there is several of them. Kibana version 7.7 OS: Ubuntu 16.04 It is a single node Kibana with 4 CPUs 8GB RAM. ES cluster: 13 nodes, Hot-Warm-Cold architecture 2x hot node, 4x warm node, 2x cold node, 3x master, 1x coordinator 1x ingest Cluster heap 26.2 GB / 64.0 GB Indices: 328 Shards: 1296 Documents: 3,051,043,815 Storage 4TB Is there anything we can do to improve performance of SIEM app? I'll be happy to provide additional details if necessary. ---
**Author:** ![Frank\_Hassanabad](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/frank_hassanabad/32/49255_2.png) [@Frank\_Hassanabad](https://discuss.elastic.co/u/Frank_Hassanabad)\ **Post date:** [May 26, 2020, 3:32pm UTC](https://discuss.elastic.co/t/kibana-siem-app-performance/234358/2 "2020-05-26T15:32:48Z")
What's your time range looking like? ![Screen Shot 2020-05-26 at 9.28.30 AM](https://us1.discourse-cdn.com/elastic/original/3X/e/2/e2c7e0288a79225752851d031c5bf04bafec20b2.png) Sometimes when there are just very large volumes of data it is going to be slow unless you cut back on your time range. ---
**Author:** ![j91321](https://avatars.discourse-cdn.com/v4/letter/j/7ea924/32.png) [@j91321](https://discuss.elastic.co/u/j91321)\ **Post date:** [May 26, 2020, 4:00pm UTC](https://discuss.elastic.co/t/kibana-siem-app-performance/234358/3 "2020-05-26T16:00:12Z")
Normally I keep the time range at 24h. It usually returns about 150 signals. ---
**Author:** ![willemdh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/willemdh/32/16922_2.png) [@willemdh](https://discuss.elastic.co/u/willemdh)\ **Post date:** [May 27, 2020, 6:03am UTC](https://discuss.elastic.co/t/kibana-siem-app-performance/234358/4 "2020-05-27T06:03:39Z")
Sorry to jump in on this, but I also think performance in the SIEM Kibana app is not optimal. We only use the builtin rules. ---
**Author:** ![j91321](https://avatars.discourse-cdn.com/v4/letter/j/7ea924/32.png) [@j91321](https://discuss.elastic.co/u/j91321)\ **Post date:** [May 27, 2020, 8:46am UTC](https://discuss.elastic.co/t/kibana-siem-app-performance/234358/5 "2020-05-27T08:46:01Z")
Just as a test I recreated the detections tab (using .siem-signals-[space]-\*) as a dashboard and it gets loaded instantly even for large time ranges. I understand the SIEM app does a lot more with the interface, just wanted to point out that it's more like SIEM app issue and not our Elasticsearch cluster having problems. ---
**Author:** ![probson](https://avatars.discourse-cdn.com/v4/letter/p/e47c2d/32.png) [@probson](https://discuss.elastic.co/u/probson)\ **Post date:** [May 29, 2020, 8:51am UTC](https://discuss.elastic.co/t/kibana-siem-app-performance/234358/6 "2020-05-29T08:51:18Z")
We found that the SIEM app was very slow, this was when accessing though Remote Desktop. When accessing from a laptop with a modern i7 it is much faster. ---
**Author:** ![tudor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor/32/3753_2.png) [@tudor](https://discuss.elastic.co/u/tudor)\ **Post date:** [May 29, 2020, 9:34am UTC](https://discuss.elastic.co/t/kibana-siem-app-performance/234358/7 "2020-05-29T09:34:48Z")
Ján, if I understand correctly, it seems like the Elasticsearch cluster is doing well, and it seems more of an issue with the Kibana server rather than the browser side, because the graphql calls are slow. What is the CPU usage do you see for the Kibana process (node.js)? It's single threaded so I'm trying to understand if it might be getting saturated. > At first I was thinking detection engine is causing this since we use a 400+ rules. So I have increased workers value to 100. Which workers setting are you referring to here? ---
**Author:** ![j91321](https://avatars.discourse-cdn.com/v4/letter/j/7ea924/32.png) [@j91321](https://discuss.elastic.co/u/j91321)\ **Post date:** [May 29, 2020, 9:49am UTC](https://discuss.elastic.co/t/kibana-siem-app-performance/234358/8 "2020-05-29T09:49:28Z")
Kibana process CPU based on monitoring is 2.31 on 5m average and 2.06 on 15m. If I look directly with htop on server the Kibana process jumps between ~170% up to ~320% (VM has 4 CPUs). The workers setting I mentioned is `xpack.task_manager.max_workers` as discussed in github issue [54697](https://github.com/elastic/kibana/issues/54697). ---
**Author:** ![j91321](https://avatars.discourse-cdn.com/v4/letter/j/7ea924/32.png) [@j91321](https://discuss.elastic.co/u/j91321)\ **Post date:** [May 29, 2020, 10:24am UTC](https://discuss.elastic.co/t/kibana-siem-app-performance/234358/9 "2020-05-29T10:24:26Z")
I have figured out where the problem is. We use different spaces with different SIEM settings for multi-tenancy. The space where the issue occurs has the `siem:defaultIndex` set to custom value, for indices that use ECS template (but not the original, `winlogbeat-* auditbeat-*` etc.) It works on our custom indices for `sophos-*, checkpoint-*` etc. Unfortunately one of those indices suffered a field explosion because of an error in Logstash parsing and now when I refreshed the Index pattern I see that it has around 8000 fields. As soon as I removed this pattern from `siem:defaultIndex` the SIEM app is responsive again. Not sure why detection tab was especially so badly affected by this, none of the detection rules actually used the pattern where field explosion occurred, but it seems to be the root cause. The `graphql ` values in logs are still the same, so it probably has nothing to do with this. ---
**Author:** ![tudor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tudor/32/3753_2.png) [@tudor](https://discuss.elastic.co/u/tudor)\ **Post date:** [May 29, 2020, 10:52am UTC](https://discuss.elastic.co/t/kibana-siem-app-performance/234358/10 "2020-05-29T10:52:11Z")
Thanks for getting back to us, and I'm glad it got better. One question about the 400 rules: are they 400 in total, not per space, right? I'm asking because you have mentioned spaces and if you have the same rule in two spaces, that counts as two rules that are executed independently. Given the high CPU usage of the Kibana server, it might still make sense to add multiple Kibana instances to avoid having gaps in detection. The Kibana task manager, which powers the detections rules, is able to scale by adding multiple instances. Adding a second Kibana on the same 4 vCPUs server might be a good way to start, because the nodejs process is otherwise single-threaded. ---
**Author:** ![j91321](https://avatars.discourse-cdn.com/v4/letter/j/7ea924/32.png) [@j91321](https://discuss.elastic.co/u/j91321)\ **Post date:** [May 29, 2020, 11:01am UTC](https://discuss.elastic.co/t/kibana-siem-app-performance/234358/11 "2020-05-29T11:01:43Z")
> One question about the 400 rules: are they 400 in total, not per space, right? I'm asking because you have mentioned spaces and if you have the same rule in two spaces, that counts as two rules that are executed independently. Yes I'm aware that it counts as separate rules. It's 21 rules in the space that had the performance problem, and another 481 rules in default space. (Most of these are from sigma project) > Given the high CPU usage of the Kibana server, it might still make sense to add multiple Kibana instances to avoid having gaps in detection. The Kibana task manager, which powers the detections rules, is able to scale by adding multiple instances. Adding a second Kibana on the same 4 vCPUs server might be a good way to start, because the nodejs process is otherwise single-threaded. I realized this as I was debugging the issue and has already started deploying additional Kibana node, but I'll also look into the possibility to run multiple instances on the same node if the HW allows it. Thank you for your help Tudor. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 26, 2020, 11:01am UTC](https://discuss.elastic.co/t/kibana-siem-app-performance/234358/12 "2020-06-26T11:01:56Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.