Huge 4 minute delay between adding a document and it showing up in Kibana

Elastic Stack Elasticsearch
1

Hi all,

When logstash posts even a single document to ElasticSearch, it takes about 4 minutes for it to show up in Kibana.

I installed the ELK stack (ES5.3) on a fresh Debian install on old hardware.
The machine has 8Gb RAM, a 2 Ghz AMD Athlon CPU and a 7200RPM drive (using the deadline scheduler because it performs best on this particular hardware.)

I've monitored the calls from LS to ES and those get sent pretty much instantly after LS receives the message.
I've also done some manual (CURL) posting through ES's API using "refresh=wait_for" switch.
The call returns in about 3 to 4 seconds.

However, before the data appears in Kibana takes about 4 minutes!
(Yes, I'm performing a wildcard search in the discover screen, every 5 minutes)

I read that data should be searchable after about 1 second on decent hardware.
My hardware may not be great but it seems to cope quite well. No bizarre CPU or disk usage.

My cluster state is yellow (there's only one instance of ES in "the cluster")
The memory locking warnings I used to have were solved and everything seems peachy.

But why this thing takes so long is beyond me.

I haven't been able to enable the index slow log. For some reason I can't make that work.

Any pointers / suggestions?

Thanks in advance!

2

That seems pretty long, but much better than your four minutes. You might be better off asking on the Kibana forums though because that 3-4 seconds is the elasticsearch time. Everything else is "something else" time. Maybe Kibana? I'm not sure.

3

Thanks Nik, I'll try that too. Although the search on Kibana seems to be lightning fast when I filter etc...

4

Oh, and would you have any pointers / example on how to enable the slowlog?

Through the API apparently isn't an option (anymore). I keep getting errors like:

curl -XPUT 'localhost:9200/_cluster/settings' -H 'Content-Type: application/json' -d '{ "transient" : { "index.indexing.slowlog.threshold.index.warn": "10s", "index.indexing.slowlog.threshold.index.info": "5s", "index.indexing.slowlog.threshold.index.debug": "2s", "index.indexing.slowlog.threshold.index.trace": "500ms", "index.indexing.slowlog.level": "info", "index.indexing.slowlog.source": 1000 } }';

{
"error": {
"root_cause": [{
"type": "illegal_argument_exception",
"reason": "transient setting [index.indexing.slowlog.level], not dynamically updateable"
}],
"type": "illegal_argument_exception",
"reason": "transient setting [index.indexing.slowlog.level], not dynamically updateable"
},
"status": 400
}

I've tried adding this to the log4j options too, but I don't get errors, nor a slowlog.
Also I've tried replacing the "index." part for the actual index name I'm interested in, yet to no avail.

5

I've checked some moer and I doubt it's Kibana.

When I perform the same query Kibana does, indeed I get no results from ES:

curl -XPOST 'localhost:9200/_msearch' -H 'Content-Type: application/json' -d '
{"index":[".kibana"],"ignore_unavailable":false,"preference":1493670146168}
{"query":{"bool":{"must_not":[{"match_all":{}}]}}}
';

I've tried without the "preference" too and also with ignore_unavailable: true

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.