The aggregations key is missing from the response, check your permissions for this request

[Akhil_K]

I am pretty new to ELK stack. We monitor a particular server health using ELK stack. We have Metricbeat which sends statics and based on that we have index created in Elastic. Kibana visualization is created based on this. But from last couple of days we get below error in Kibana.

Observations :

1. We validated Index Management to check if we have data. We could find that there is no issue in streaming as we continue to receive data.
   

   

2. When tried to view the index from Discover, got below Request Timeout Error.
   

3. Any query we run in Dev Tools returns Internal Server Error, as like below.
   

   

We have ELK setup in dockerised env, ie, seperate docker for Elastic, Logstash and Kibana. When I restart the Kibana container alone, Kibana is back normal. But after sometime, I get this error again.

Tried below solution, but nothing positive.

Kibana The aggregations key is missing from the response ERROR

Hi i just followed this tutorial: https://www.elastic.co/guide/en/elastic-stack-get-started/current/get-started-elastic-stack.html To set up a little ELK-stack on my Ubuntu VM. I used metricbeats so that i could monitor my VM as practice. My Kibana dashboard is showing me: The request for this panel failed. The aggregations key is missing from the response on all views, so no data is visible. Any idea what could cause this? No Kibana logs whatsoever. [kibanakak]

What could be the reason. We are stuck with this for sometime. Thanks in advance

[mattkime (Matthew Kime)]

Can you look at the elasticsearch and kibana logs when you run your simple test query above? You might use https://gist.github.com/ to post long logs.

Its likely that either elasticsearch is having problems or kibana is having trouble connecting to elasticsearch.

[Akhil_K]

Elasticsearch log looks fine as I don't see any issue/error in those. However, in Kibana logs I could notice few logs with Error and statuscode 404 as shared below :

Kibana error logs

There weren't any configuration changes made to my system recently and everything was running fine till last few days. When I pulled the Kibana log, i could figure out only these logs which was repeating. And when we do a container restart, Kibana visualization will be up and these logs wont appear until next time we get issue in visualization. At same time, Elasticsearch logs are consistent and there are no errors in them.

[mattkime (Matthew Kime)]

Can you share your kibana config? I'm curious what might be causing this - Error: Specified object should be an Object or Array

[Akhil_K]

PFB the kibana config :
server.name: kibana
server.host: "0"
elasticsearch.url: http://elasticsearch:9200
xpack.monitoring.ui.container.elasticsearch.enabled: true

As I said earlier, this was the config for sometime and there wasn't any issues so far, until recently.

[mattkime (Matthew Kime)]

What do you see in the browser console when the error occurs? which browser and version are you using?

I have a hunch that this is a networking problem but I need to gather info to verify that.

[Akhil_K]

The console gives Gateway Timeout error. PFB the console view :

[mattkime (Matthew Kime)]

I don't think that 504 error is related to the request immediately above. Can you show the network tab? Which version of the elastic stack are you running? Which license are you using?

[Akhil_K]

PFB the network tab :

PFB the network response :

_bulk_get

fields?index=indexname-*

data

Elastic stack version is 6.4.3, and its a Basic license we are holding.

[mattkime (Matthew Kime)]

Can you verify that the info you're attempting to view is available in Discover?

Thanks for your patience, I understand the importance of resolving this.

[Akhil_K]

No. I get "Discover: Request Timeout after 30000ms" error stack as attached below.

Apologies for the delay in response. Was in a travel.

[mattkime (Matthew Kime)]

Lets work on this error for now. Can you find the request timeout in the dev tools network tab? I'd like to see the details of which request was made. Its also worth checking kibana and elasticsearch logs for errors that occurred at the same time.

[Akhil_K]

PFB the network tab during the timeout :

The request made is to below api :
/elasticsearch/_msearch

I couldn't fetch the logs as it would take some time but I guess it would any of 3 logs which we discussed earlier. I am sharing that again below :

Error logs from Kibana shared earlier

[mattkime (Matthew Kime)]

Can you click on the _msearch query so we can see the headers and params that are passed?

I do need you to look at the kibana and elasticsearch logs to look for corresponding errors.

[Akhil_K]

Thanks for the response. PFB the headers :

While checking logs, I could not find anything except for the ones shared earlier. That 3 logs are repeating.

[mattkime (Matthew Kime)]

Thanks, while thats the correct view, I need to see the Request Payload portion. I should have specified the first time around.

Request number 2 - Could you try this with dev tools and tell me what you get? GET _cluster/health

[Akhil_K]

PFB the Request Payload portion :

Also PFB the response from Devtools when hitting GET _cluster/health :

[mattkime (Matthew Kime)]

The fact that we can't get a response from GET _cluster/health is concerning. What privileges does your user for kibana have? Is there another account with full privileges? Either your cluster has a problem or kibana is having trouble connecting to your cluster.

Can you connect to your es cluster directly? curl -X GET "localhost:9200/_cluster/health?wait_for_status=yellow&timeout=50s&pretty" - changing address and port as appropriate.

Can you do the same from the machine that kibana is running on?

Documentation on the _cluster/health endpoint - https://www.elastic.co/guide/en/elasticsearch/reference/6.4/cluster-health.html

[Akhil_K]

Below is what I get when trying to connect es cluster directly :

We have the basic privilege for Kibana. How do we narrow down to confirm if the problem is with cluster or with connection trouble between Kibana and cluster?

[mattkime (Matthew Kime)]

You ran the command from the browser. Can you try running it from the command line?

Can you run that same command directly from the server instance running elasticsearch?

Can you check the elasticsearch logs to see if there's a matching error in the logs?