Kibana search 7 days throws error

Elastic Stack Elasticsearch
1

Hi i am getting this error (Discover throws error: "An error occured with your request. Reset your inputs and try again") in kibana when i try to do a search of time length 7 days or longer.

I modified the request_timeout setting in kibana.yml and restarted kibana
request_timeout: 600000

Still throwing the same error.

There is no errors in the elasticsearch client node that kibana is connected to.

I checked a bunch of the data nodes and i see GC happening during the time of the long search. When i run a 7 day query, is elasticsearch trying to load all 7TB of indices data into memory and that is why the kibana query is failing?

I have 400GB of heap memory and 37TB of indices data. Are there any tweaks that i can do to make large queries work or am i limited to querying a smaller subset of data.

Thanks.

2

What version of KB and ES?

3

both latest and greatest

kibana 4.1.1
ES1.7.0

4

How many Elasticsearch nodes are you running?

Also, try running the same query on Elasticsearch itself (using cURL or Marvel's Sense or another HTTP Client) and see what the error in the response or in the server logs is.

5

8 elasticsearch data nodes.

what is the equivalent marvel sense query of showing * in last 7 days?

6

That query will depend on how you have set up your data across indices. Do you have a single index for all your data, or are you running time-series indices? If you are running time-series indices, do you have an index per day, week, or month? What is the pattern of your index name?

If you are unsure about any of the above questions, could you run the following in Marvel Sense and paste the output (if you have a huge number of indices, just paste the bits of the output relevent to what you are using in Kibana.

GET _cat/indices?v
7

I use the logstash created daily indices. with logstash-%Y.%m.%d

8

in which case you could just run the following in Sense to get search the last 7 days:

GET logstash-2015-07-21,logstash-2015-07-20,logstash-2015-07-19,logstash-2015-07-18,logstash-2015-07-17/_search

BUT

I would first try searching just one of your indices and see if it times out:

GET logstash-2015-07-21/_search

And then if that works add a second index and so on until you have all 7. If it errors then paste the response here (you should also check the Elasticsearch server logs when it fails for error logs).

If the response or error in the server logs is long, please create a gist containing them and link to it here.

9

This is the marvel sense query i ran which i assume is the equivalent of running * search on kibana

GET logstash-2015.07.21,logstash-2015.07.20,logstash-2015.07.19,logstash-2015.07.18,logstash-2015.07.17,logstash-2015.07.16,logstash-2015.07.15/_search
{
"query": {
"match_all": {
}
}
}

Result came back really fast
"took": 2969,
"timed_out": false,
"_shards": {
"total": 56,
"successful": 56,
"failed": 0
},
"hits": {
"total": 9140358688,
"max_score": 1,

Might the slowness be because kibana by default does all searches against the message field?

10

hmmm, interesting, I was expecting that to fail since it fails in Kibana.

I am assuming you are using Kibana 4 but correct me if I am wrong. In K4 you can see the request that is being run on Elasticsearch on every visualisation. To do this you need to click the up arrow at the bottom of the visualisation:

example.png1272×1098 90.1 KB

And then click request. You should see something like this:
example2.png1276×1102 131 KB

If you copy that request JSON you can then run that in Sense and see if it reproduces your error.

11

The queries work when running in sense.
I am not longer sure this is an elasticsearch issue since the sense queries work.

I tried to open a github issue and discuss topic in kibana forum but both were rejected.

Here is my gist with the results of the sense queries.
https://gist.github.com/allenmchan/f4fed82d7914c3c5171a

12

Could you explain the visualization you are trying to run in Kibana? What you want it to show and how you created it?

I suspect the majority of the time is taken in by it trying to highlight all fields. I'm not an expert with Kibana so would be good to know how you created the visualisation in the Kibana UI so I can understand why the highlight bit of the request is there.

13

I am not doing a visualization. Kibana has a discover tab that just does searches. By default i think highlight is turned on for all searches. All i am doing is opening kibana, selecting 7 days for timeframe and * search

14

@allenmchan Do I have this right? Kibana thinks the request failed, the server logs indicate that the requests were fine, and sense executes the requests fine.

Can you use Chrome to access Kibana and check out the network debugging panel? If you search through the requests for the /elasticsearch/_msearch I'd like to see the response that kibana is getting (it is probably different from what sense is showing).

15

This is embarassing but Spencer's comment lead me down to the troubleshooting process that found the issue. I was using nginx was a proxy to get through to kibana. Nginx was timing out. I only found this when i was looking at the network debugging console.

Thanks for everyone's attention on this. My apologies it turned out being my issue.

16

Glad to hear you tracked it down! :blush: