Couldn't refresh index and now times out with "Bad Request"


(Peter) #1

I believe this is an Elasticsearch problem although it manifests itself in Kibana.

I'll apologize upfront -- this is my first post. I'll try and provide what is needed.

We're using ELK to gather the logs of about 1,500 machines including F5s and ESXi. They are sorted into 8 different daily indexes totaling 6.6B documents over the 30 day retention. In monitoring everything is green. The indexes, nodes, and Kibana. I've been unable to refresh the index for our default index which is logstash. It is by far our largest index as well. At about 100m documents daily and a data size of around 220GB. There are 5 shards and one replica. We have 10 data nodes with 5 at each site and a fast pipe between them. When I would try to refresh it within Kibana it would timeout.

Yesterday I upgraded it via Puppet from version 5.4.1 to 5.5.0. Now I can't access the data in the logstash indexes either. When I hit Kibana it just spins and then errors. I can see the data in other indexes. The error looks like this:

Error: Request to Elasticsearch failed: "Bad Request"
at http://kibana.example.com/bundles/kibana.bundle.js?v=15382:229:4009
at processQueue (http://kibana.example.com/bundles/commons.bundle.js?v=15382:38:23621)
at http://kibana.example.com/bundles/commons.bundle.js?v=15382:38:23888
at Scope.$eval (http://kibana.example.com/bundles/commons.bundle.js?v=15382:39:4619)
at Scope.$digest (http://kibana.example.com/bundles/commons.bundle.js?v=15382:39:2359)
at Scope.$apply (http://kibana.example.com/bundles/commons.bundle.js?v=15382:39:5037)
at done (http://kibana.example.com/bundles/commons.bundle.js?v=15382:37:25027)
at completeRequest (http://kibana.example.com/bundles/commons.bundle.js?v=15382:37:28702)
at XMLHttpRequest.xhr.onload (http://kibana.example.com/bundles/commons.bundle.js?v=15382:37:29634)

The Kibana node is also an Elasticsearch node but doesn't hold data. I don't see anything of significance in either the Kibana log or the ES log with log level set to info. Here is a Kibana timeout log message:

{
    "type":"response",
    "@timestamp":"2017-07-14T15:35:07Z",
    "tags":[],
    "pid":22484,
    "method":"post",
    "statusCode":400,
    "req":{
        "url":"/api/console/proxy?path=_mapping&method=GET",
        "method":"post",
        "headers":{
            "host":"kibana.example.com",
            "connection":"keep-alive",
            "content-length":"0",
            "accept":"text/plain, */*; q=0.01",
            "origin":"http://kibana.example.com",
            "kbn-version":"5.4.1",
            "user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit /537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36",
            "referer":"http://kibana.example.com/app/kibana",
            "accept-encoding":"gzip, deflate",
            "accept-language":"en-US,en;q=0.8"
        },
        "remoteAddress":"192.168.59.3",
        "userA gent":"192.168.59.3",
        "referer":"http://kibana.example.com/app/kibana"
    },
    "res":{
        "statusCode":400,
        "responseTime":39,
        "contentLength":9
    },
    "message":"POST /api/console/proxy?path=_mapping&method=GET 400 39ms - 9.0B"
}

The entire ELK stack is at 5.5.0.

Thanks,

Peter


(Chen Jian) #2

I have encounter the same issue after upgrading to ES 5.5 :frowning:.


(Terry Wade) #3

HI

Was there a fix for this? It seems to have happened with our upgrade from 5.2.1 to 5.5.1.

Terry


(David Pilato) #4

I moved it to #kibana in case anyone there has an idea.


(blsonne) #5

Happening to me as well... upgraded from 5.3.1 to 5.5.1 and now nothing I do allows me to configure a default index pattern without getting the following error:

Error: Request to Elasticsearch failed: "Bad Request"
at http://elasticsearch-05:5601/bundles/kibana.bundle.js?v=15405:229:4009
at processQueue (http://elasticsearch-05:5601/bundles/commons.bundle.js?v=15405:38:23621)
at http://elasticsearch-05:5601/bundles/commons.bundle.js?v=15405:38:23888
at Scope.$eval (http://elasticsearch-05:5601/bundles/commons.bundle.js?v=15405:39:4619)
at Scope.$digest (http://elasticsearch-05:5601/bundles/commons.bundle.js?v=15405:39:2359)
at Scope.$apply (http://elasticsearch-05:5601/bundles/commons.bundle.js?v=15405:39:5037)
at done (http://elasticsearch-05:5601/bundles/commons.bundle.js?v=15405:37:25027)
at completeRequest (http://elasticsearch-05:5601/bundles/commons.bundle.js?v=15405:37:28702)
at XMLHttpRequest.xhr.onload (http://elasticsearch-05:5601/bundles/commons.bundle.js?v=15405:37:29634)

Elasticsearch has been nothing but a fight over the last couple years, and I'm ready to give up on this nonsense.


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.