Terribly slow kibana query

Elastic Stack Kibana
1

I have around 2GB of events getting accumulated every day. I use curator to get rid of data every 5 days.

Now, the problem is it takes long time to execute search in kibana. I am using docker to run ELK . When I do docker stats , I can see that only 3 GB of memory is getting used when I have 64GB available .

image
image.png1895×99 4.98 KB

I check the heap : It shows the max heap is 16GB (which according to ergonomics algorithm is correct .

According to the algorithm , The max heap is 1/4 of the total and the initial heap is 1/64 of the total . This is what i can see :

 java -XX:+PrintFlagsFinal -version | grep -iE 'HeapSize|P                                                                             ermSize|ThreadStackSize'
 intx CompilerThreadStackSize                   = 0                                                                                                                {pd product}
uintx ErgoHeapSizeLimit                         = 0                                                                                                                {product}
uintx HeapSizePerGCThread                       = 87241520                                                                                                         {product}
uintx InitialHeapSize                          := 1054867456                                                                                                       {product}
uintx LargePageHeapSizeThreshold                = 134217728                                                                                                        {product}
uintx MaxHeapSize                              := 16848519168                                                                                                      {product}
 intx ThreadStackSize                           = 1024                                                                                                             {pd product}
 intx VMThreadStackSize                         = 1024                                                                                                             {pd product}

I have increased the shards to 15 and replicas to 5 for two indices that I have. But no change in memory usage.

How to make the search results faster ?

2

Please don't post pictures of text, they are difficult to read, impossible to search and some people may not be even able to see them :slight_smile:

How long?

What is the output from GET _cat/indices?

3

Thanks for responding back.
It takes around 15-20 seconds .

I have got two indices . I just deleted the indices as I was playing with the shards and replicas. Anyways , here are the indices :

yellow open filebeat-failurelogs xctk6SElTImQipMLiTNDqw 15 5       6 0  78.4kb  78.4kb
green  open .kibana              aJQ8FbAbQU-S6EAfvN30Uw  1 0       2 0  15.8kb  15.8kb
yellow open filebeat-successlogs k3YK20_wRFO7MSd6WW-qQA 15 5 2615032 0 618.9mb 618.9mb

Both the indices collect no more than 2 Gb per day.

Should I update the Xms and Xmx in JVM options of ElasticSearch to around 70% of physical memory i.e around 40 GB ?

Do I need to do anything for Logstash and Kibana end ?

4

You really only need 1 primary shard for 2GB of data, anymore is just wasteful and likely to cause slower response times, as the heap will be taken up managing mostly empty shards.

1 Like
5

Okay. Glad to know .

It will collect 2 GB per day , but will accumulate for 5 days to around 10-11 GB before the curator removes some of the data.

In that case is going with the default of 3 shards 2 replicas okay ?

6

Let me go with 3 shards 2 replicas - the default setting and 40 GB max and min heap.
i.e
-Xms40g
-Xmx40g

I will get back to inform the community of my findings.

7

If you are only going to have that little data in the cluster, set the heap (Xms and Xmx) to 4GB. Then use a single primary shard per daily index. If you have more than 1 node, set the number of replica shards to 1 (or possibly 2 if you have 3 nodes in the cluster).

1 Like
8

It might be easier to keep things to a single thread, as you also have this one going too - Increase shards and replicas in new Index

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.