Elasticsearch search query performance

Elastic Stack Elasticsearch
1

Hello ,

I have elasticsearch 5.2 running on a big VM :
128 GB RAM
8 CPU

As mentioned in the documentation , I set Heap to slightly less than half to 60 GB.
The problem : In kibana, I want to display data tables with 80000 lines and 50 columns ( sometimes even more)
the result : either so slow or no result ( I changed timeout conf in kibana).
So recently , I added another node but no major change in performance.

The question : How to improve performance and boost search query speed by any means possible ? ( configurations , hardware ...)

best regards,
Moka

2

What type of storage do you have? How much data do you have? How many are nfices and shards?

3

I have 500 GB storage on both master and slave.
I have 145 gb data respecting the rule of having less than 80% of total storage.
I have around 50 indices and 650 shards.

4

That sounds like s lot of shards for that data volume but should be fine. It is. Do generally recommended to keep the heap below 32GB in size. Having reread the post I however suspect it is the query you are running that is extremely expensive and used a lot of heap. A data table with 50 columns and 80000 rows sounds excessive and not really what Kibana was designed for.

How do you even use such a massive data table? Do you have any stats on how long the query takes to run in Elasticsearch versus how long the browser takes to parse and process such large result?

5

The purpose of creating massive tables : to give raw data to other teams so they can do whatever they want with them .
the perk of using kibana is that data is updated daily and also ,so no one query the database .
the data source is : an sql view and injected into the index via logstash :
the input plugin in logstash is the connector mysql .
for the comparison between elasticsearch and browser is not done yet.
I will keep you updated in the coming days

6

That kind of query is likely to be expensive no matter where you run it so you may need more resources or perhaps store preaggregated values if that is possible.

7

I understand , so maybe the solution is adding more nodes .
my question is the following :
I ve read in the docs ,that adding nodes improves capacity .
what does that mean exactly ?
Are we talking speed boost or simply storage capacity ?

8

Both. Adding nodes mean adding system resources, which either allows to to add storage or increase performance.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.