Kibana parsing_exception on Unknown key for a START_ARRAY

Elastic Stack Kibana
1

I am running ELK 5.0.0-alpha5 for logging. Everything was running beautifully until I upgraded from alpha1 to alpha 5. Logstash is indexing logs into Elasticsearch and I can view them via the Elasticsearch API. Unfortunately Kibana now gives the following error when searching:

Error: [parsing_exception] Unknown key for a START_ARRAY in [stored_fields]., with { line=1 col=577 }
at respond (http://X.X.X.X:5601/bundles/kibana.bundle.js?v=13499:22:27984)
at checkRespForFailure (http://X.X.X.X:5601/bundles/kibana.bundle.js?v=13499:22:27212)
at http://X.X.X.X:5601/bundles/kibana.bundle.js?v=13499:10:18968
at processQueue (http://X.X.X.X:5601/bundles/commons.bundle.js?v=13499:38:23627)
at http://X.X.X.X:5601/bundles/commons.bundle.js?v=13499:38:23894
at Scope.$eval (http://X.X.X.X:5601/bundles/commons.bundle.js?v=13499:39:4619)
at Scope.$digest (http://X.X.X.X:5601/bundles/commons.bundle.js?v=13499:39:2359)
at Scope.$apply (http://X.X.X.X.142:5601/bundles/commons.bundle.js?v=13499:39:5037)
at done (http://X.X.X.X:5601/bundles/commons.bundle.js?v=13499:37:25027)
at completeRequest (http://X.X.X.X:5601/bundles/commons.bundle.js?v=13499:37:28702)

I have tried removing the the .kibana index, restarting kibana, and rebuilding the index. I even tried reinstalling kibana. But I continue to get the same error. Any ideas?

2

Hi,

Which Kibana page are you on? Most Kibana pages have a small ^ icon that will reveal the HTTP API request being made from Kibana to Elasticsearch. Could you find this and paste the HTTP API request here? That will help us figure out where this error is coming from.

Thanks!

Shaunak

3

I don't see a button. See the attached screenshot. The URL of Kibana does update too:
http://X.X.X.X:5601/app/kibana#/discover?_g=(refreshInterval:(display:Off,pause:!f,value:0),time:(from:now-24h,mode:quick,to:now))&_a=(columns:!(_source),index:'logstash-',interval:auto,query:(query_string:(analyze_wildcard:!t,query:'')),sort:!('@timestamp',desc))

kibana.png1348×380 20.1 KB

I did notice that if I go to the kibana console and do a query for:
GET _search
{
"query": {
"match_all": {}
}
}

I get the results:
Error connecting to 'https://X.X.X.X/elasticsearch/_search':
Client request error: unable to verify the first certificate

If I paste that URL in the browser, Elasticsearch does return the appropriate json result. I do have the kibana configuration set to disregard the validity of SSL certificates.

4

Alright, I guess the button isn't shown when there's an error. There's another way we can figure out what query is being sent to Elasticsearch. Can you open the Developer Console for your browser and look at the network requests. In that there should be a failing one. Can you paste the request's URL and body?

5

There is and it is strange:
Request URL:http://X.X.X.X:5601/elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1472762725907
Request Method:POST
Status Code:400 Bad Request
Remote Address:X.X.X.X:5601

What is strange is that this is setup on an intranet and kibana is on a different server than elasticsearch. The IP in the bad request is the IP for the kibana server, not the elasticsearch server. I went back and double checked the kibana.yml and verified that it did have the correct IP under the elasticsearch.url variable and the file has correct permissions. It is almost as if kibana is not reading the config and using default settings.

6

The IP in the bad request (what you are showing as X.X.X.X) should be the IP for the Kibana server. The Kibana server proxies the search request through to Elasticsearch. So that request URL looks good.

Would you mind grabbing the body of that request and pasting it here?

7

General
Request URL:http://X.X.X.142:5601/elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1472763385917
Request Method:POST
Status Code:400 Bad Request
Remote Address:X.X.X.142:5601
Response Headers
cache-control:no-cache
connection:close
content-encoding:gzip
content-type:application/json; charset=UTF-8
date:Thu, 01 Sep 2016 20:56:36 GMT
kbn-name:kibana
kbn-version:5.0.0-alpha5
server:Apache/2.4.6 () OpenSSL/1.0.1e-fips mod_fcgid/2.3.9
Transfer-Encoding:chunked
vary:accept-encoding
Request Headers
Accept:application/json, text/plain, /
Accept-Encoding:gzip, deflate
Accept-Language:en-US,en;q=0.8
Connection:keep-alive
Content-Length:674
content-type:application/x-ldjson
Host:x.x.x.142:5601
kbn-version:5.0.0-alpha5
Origin:http://x.x.x.142:5601
Referer:http://x.x.x.142:5601/app/kibana
User-Agent:Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
Query String Parameters
timeout:0
ignore_unavailable:true
preference:1472763385917
Request Payload
{"index":["logstash-2016.09.01"],"ignore_unavailable":true}
{"size":500,"sort":[{"@timestamp":{"order":"desc","unmapped_type":"boolean"}}],"highlight":{"pre_tags":["@kibana-highlighted-field@"],"post_tags":["@/kibana-highlighted-field@"],"fields":{"":{}},"require_field_match":false,"fragment_size":2147483647},"aggs":{"2":{"date_histogram":{"field":"@timestamp","interval":"30s","time_zone":"America/New_York","min_doc_count":1}}},"query":{"bool":{"must":[{"match_all":{}},{"range":{"@timestamp":{"gte":1472762487548,"lte":1472763387548,"format":"epoch_millis"}}}],"must_not":[]}},"stored_fields":[""],"_source":true,"script_fields":{},"fielddata_fields":["@timestamp"]}

Preview
{error: {,…}, status: 400}
error:{,…}col:540line:1reason:"Unknown key for a START_ARRAY in [stored_fields]."root_cause:[{type: "parsing_exception", reason: "Unknown key for a START_ARRAY in [stored_fields].", line: 1,…}]type:"parsing_exception"status:400

Response
{"error":{"root_cause":[{"type":"parsing_exception","reason":"Unknown key for a START_ARRAY in [stored_fields].","line":1,"col":540}],"type":"parsing_exception","reason":"Unknown key for a START_ARRAY in [stored_fields].","line":1,"col":540},"status":400}

8

This sounds a lot like this issue: https://github.com/elastic/kibana/issues/7828. I'd suggest either building Kibana from master OR waiting for the next release where this issue is fixed. The next release should be coming out very shortly.

9

Good catch, that does seem like the same issue. I will take your advice and wait for the next release. Thank you for your help!