Terribly slow kibana query

I have around 2GB of events getting accumulated every day. I use curator to get rid of data every 5 days.

Now, the problem is it takes long time to execute search in kibana. I am using docker to run ELK . When I do docker stats , I can see that only 3 GB of memory is getting used when I have 64GB available .

I check the heap : It shows the max heap is 16GB (which according to ergonomics algorithm is correct .

According to the algorithm , The max heap is 1/4 of the total and the initial heap is 1/64 of the total . This is what i can see :

 java -XX:+PrintFlagsFinal -version | grep -iE 'HeapSize|P                                                                             ermSize|ThreadStackSize'
 intx CompilerThreadStackSize                   = 0                                                                                                                {pd product}
uintx ErgoHeapSizeLimit                         = 0                                                                                                                {product}
uintx HeapSizePerGCThread                       = 87241520                                                                                                         {product}
uintx InitialHeapSize                          := 1054867456                                                                                                       {product}
uintx LargePageHeapSizeThreshold                = 134217728                                                                                                        {product}
uintx MaxHeapSize                              := 16848519168                                                                                                      {product}
 intx ThreadStackSize                           = 1024                                                                                                             {pd product}
 intx VMThreadStackSize                         = 1024                                                                                                             {pd product}

I have increased the shards to 15 and replicas to 5 for two indices that I have. But no change in memory usage.

How to make the search results faster ?

Please don't post pictures of text, they are difficult to read, impossible to search and some people may not be even able to see them :slight_smile:

How long?

What is the output from GET _cat/indices?

Thanks for responding back.
It takes around 15-20 seconds .

I have got two indices . I just deleted the indices as I was playing with the shards and replicas. Anyways , here are the indices :

yellow open filebeat-failurelogs xctk6SElTImQipMLiTNDqw 15 5       6 0  78.4kb  78.4kb
green  open .kibana              aJQ8FbAbQU-S6EAfvN30Uw  1 0       2 0  15.8kb  15.8kb
yellow open filebeat-successlogs k3YK20_wRFO7MSd6WW-qQA 15 5 2615032 0 618.9mb 618.9mb

Both the indices collect no more than 2 Gb per day.

Should I update the Xms and Xmx in JVM options of ElasticSearch to around 70% of physical memory i.e around 40 GB ?

Do I need to do anything for Logstash and Kibana end ?

You really only need 1 primary shard for 2GB of data, anymore is just wasteful and likely to cause slower response times, as the heap will be taken up managing mostly empty shards.

1 Like

Okay. Glad to know .

It will collect 2 GB per day , but will accumulate for 5 days to around 10-11 GB before the curator removes some of the data.

In that case is going with the default of 3 shards 2 replicas okay ?

Let me go with 3 shards 2 replicas - the default setting and 40 GB max and min heap.
i.e
-Xms40g
-Xmx40g

I will get back to inform the community of my findings.

If you are only going to have that little data in the cluster, set the heap (Xms and Xmx) to 4GB. Then use a single primary shard per daily index. If you have more than 1 node, set the number of replica shards to 1 (or possibly 2 if you have 3 nodes in the cluster).

1 Like

It might be easier to keep things to a single thread, as you also have this one going too - Increase shards and replicas in new Index

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.