The request for this panel failed. The aggregations key is missing

Elastic Stack Beats
1

So I'm running into this problem that a few others on this board have had problems with in the past. I'm somewhat confused about the problem so maybe someone could spend a minute or two to help understand the issue or point me in a documentation direction that explains it.

I've setup my cluster, logstash, kibana and I've installed Metricbeat on a server to start ingesting data. I loaded the metricbeat template in Elasticsearch as well as loading the pre-canned visualization templates that also come with it. However several of the visualizations give me the following error:

The request for this panel failed. The aggregations key is missing from the response, check your permissions for this request.

I have the basic license installed so from what I'm reading in the docs, while X-Pack is installed, security is by default turned off. I then came across this post:

Seems that it's not a permissions issue and that Fielddata is by default set to false. I ran the code found here in 6.4.2 and my visualizations started working.

However, I've since upgraded to 6.5.1 which has now broken the visualizations again. I re-ran the code found on the page but it has not (re)fixed the problem. Which now leads me to a few questions:

First, simply how to I fix what the upgrade broke? Second, based on what is written on the fielddata page, it seems that this practice isn't a good idea or at least isn't scalable. So either why wasn't a keyword field used in the Metricbeat template OR why didn't the visualization templates warn of this problem and provide avenues of resolution?

I feel like this is an error in my understanding the software. if someone could point me in a documentation direction, I would greatly appreciate it.

2

Hello! Are you running metricbeat as root and did you run ./metricbeat setup --dashboards to load kibana dashboard again after upgrading?

3

That appears to be the case. I also did re-run the dashboard command. The panels are unfortunately still displaying the error.

root 29537 2.0 0.1 2817100 44668 ? Ssl Dec05 23:10 /usr/share/metricbeat/bin/metricbeat -c /etc/metricbeat/metricbeat.yml -path.home /usr/share/metricbeat -path.config /etc/metricbeat -path.data /var/lib/metricbeat -path.logs /var/log/metricbeat

4

Could you share your full metricbeat config file?

5 
###################### Metricbeat Configuration Example #######################

# This file is an example configuration file highlighting only the most common
# options. The metricbeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/metricbeat/index.html

#==========================  Modules configuration ============================

metricbeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
  #_source.enabled: false

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging


#============================== Dashboards =====================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here, or by using the `-setup` CLI flag or the `setup` command.
#setup.dashboards.enabled: false

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
  host: "http://SVRDNSNAME:5602"
  username: "USRNAME"
  password: "PWD"


  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

#============================= Elastic Cloud ==================================

# These settings simplify using metricbeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  # hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["SVRDNSNAME:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Procesors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

#============================== Xpack Monitoring ===============================
# metricbeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#xpack.monitoring.enabled: false

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well. Any setting that is not set is
# automatically inherited from the Elasticsearch output configuration, so if you
# have the Elasticsearch output configured, you can simply uncomment the
# following line.
#xpack.monitoring.elasticsearch:
6

Couldn't find what I was hoping to see in the config. I assumed you modified potentially the index names or had the template loading disable. But this does not seem to be the case. Did you have any different configs in the beginning?

Any chance you could start a fresh setup on the side and try it again? First run metricbeat setup and then start ingesting data and see if appears? I still assume there must be something wrong with the mapping.

7

No worries. I do appreciate your time. I was able to sort this out. Well, at least getting the dashboard panels working again. Here's what I needed to do:

flush the data:

curl -XDELETE 'http://SVRDNSNAME:9200/metricbeat-*'

Repopulate the dashboard templates:

metricbeat setup --dashboards

And finally enabling fielddata for select fields depending on the affected panel.

 PUT my_index/_mapping/_doc
    {
      "properties": {
        "DATAFIELDTOBEALTERED": { 
          "type":     "text",
          "fielddata": true
        }
      }
    }

I still don't understand why I would need to make the final change for an 'out of the box' template. Nor is it documented in the setup instructions.

8

The final setup also does not make sense to me :slight_smile: You have in the above snipped my_index, is that correct?

Could you share the content of your metricbeat template? I worry it has the wrong content inside. So in the above do not only remove the indices but also the templates.

Could you share also the logs when your Beat starts up. You should have some indications there if the template was loaded or not.

9

Sorry. The code i posted was not what I ran against my data. Here's what I ran:

PUT metricbeat-*/_mapping/doc
{
  "properties": {
    "DATAFIELDTOBEALTERED": { 
      "type":     "text",
      "fielddata": true
    }
  }
}

I ran this against the various fields the visualizations complained about.

I didn't reload the template from 6.5.1, so maybe that is an issue. I ran "GET _template/metricbeat" which returns a huge ~10K line template. That's what you want me to post? Finally, you're looking for the /var/log/metricbeat logs?

10

Ok found the issue. It has zero to do with metricbeat and centered around my output from logstash.

I was using an outdated index output string in logstash:

index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"

when I should have been outputting the data like so:

index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"

I'm not sure how/why I was using that. But I came across the issue in a "breaking changes" page:

https://www.elastic.co/guide/en/beats/libbeat/current/breaking-changes-6.0.html

I've only been messing around with Elasticstack for a few months now, so I guess I picked up that line of code from an old web blog/site post when my understanding of the software was way more immature.

Here's the code from my output.conf file on logstash

output {
  elasticsearch {
    hosts => ["SVRDNSNAME:9200"] 
#    sniffing => true
    manage_template => false
    # index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.