Performance issues that make no sense

Elastic Stack Kibana
1

Hi,

I've been working with ES 5.2.2 for the past few months and I can't seem to make searches run fast enough. They even timeout quite a bit. My set up:

I have a
1 dedicated Logstash node
3 master eligible ES nodes (8GB available RAM, I set half of it, 4GB, in the JVM file)
1 coordinating node with kibana on it

My sharding is 1 primary with 2 replicas.

So when I search past 12 hours, I get a timeout which seems ridiculous because I only have about 70k documents at 160mb each day/index so you'd think things would be going fast enough but they aren't.

I've done a ton of reading on how to make things faster but nothing seems to work. As a side question, is m_lockall automatically set in the YAML file in ES 5.X, or I need to explicitly set it?

Thanks a bunch!!

2

What type of queries are you running? How much CPU does each node have? What kind of storage do you use? Do you have monitoring installed so you can get some clues from that?

3

A basic search all (*) query is what I'm referring to, but it's only a few
10k of documents that it's searching through... sadly i don't have
monitoring installed and can't install it because I've already used up the
trial. These are virtual machines using ssds and about half of the storage
is used up by past indexes. Can those past indices have an impact on
searches that don't even look at those shards? and I'll have to check on
the CPU.. what is a lower limit of the cpus? Does it have that strong of an
impact on performace?

4

Can you show your exact query?

Can you try with just:

GET _search?size=1

And share here the result?

5

{"took":6346,"timed_out":false,"_shards":{"total":20,"successful":19,"failed":0},"hits":{"total":5417057,"max_score":1.0,"hits":[{<etc.......>

6

Some more statistics that I found that could be interesting:

When now do GET _search?size=1 I get more or less:

{
"took": 12,
"timed_out": false,
"_shards": {
"total": 30,
"successful": 30,
"failed": 0
}

But what I was specifically referring to was the responsiveness of the "Discover" tab. Even just looking at the past 15 minutes the response time is:

Query Duration 1861ms
Request Duration 2538ms

and past 24 hours:

Query Duration 22710ms
Request Duration 26900ms
Hits 78176

I included the hits to show that it is so few ( and that is over one shard and 2 replicas).

Any ideas?

7

Anyone have some insight?

8

I guess the bigger question is why would the discover tab be so much slower retrieving a query than the curl command?

9

hello?

10

Knowing that there is Kibana in the middle, may be Kibana does not have enough memory or something like that?

I'm moving your question to #kibana in case my colleagues have any idea.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.