Beat Dashboards aggregating on non-aggregatable fields

[chris060986 (Christoph Birk)]

Hi,
we are using elasticsearch 5.6.2 and the beat-dashboards 5.6.2. After installing the dashboards all seems to be fine, indices are created, visualizations and dashboards are successfully stored in elastic.
But when we begin to load data into elasticsearch and open dashboards/visualization, their often occours the error "Saved field parameter is no invalid. Please select a new field"

This error only occours an string fields and when i investigate this problem i saw, that the visualizations use fields for aggregation which are not aggregatable in the index pattern.
Is it necessary to install/add a special index mapping?

[tudor (Tudor Golubenco)]

Hi,

The system.process.name field should be aggregatable indeed. When you load the Beats dashboards, they also load the index patterns, in which that field is aggregatable.

First thing that you should check is that you have the Metricbeat template loaded and correctly used for the index. Loading the template happens automatically when you use the Elasticsearch output, but needs to be loaded manually when using a different output. Is that the case for you?

You can check the mapping by calling GET /metricbeat-*/_mapping in Kibana Console. Lookup the system.process.name field and check that it's of type keyword. If it's not, it means the template wasn't loaded and you need to delete the indices and make sure the template is there before creating any documents.

If the field has the right type, try loading the Beats dashboards again.

[chris060986 (Christoph Birk)]

You are right. When I import the dashboards the fields are aggregatable. But when I receive data from logstash the mapping seems to be overwritten. Here is our logstash output config:

output {
    elasticsearch {
        hosts => ["localhost:9200"]
        sniffing => true
        manage_template => false
        index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
        document_type => "%{[@metadata][type]}"
    }
}

We also tried to set the manage template flag to true, but this doesn't change the behaviour.

Additional question: Should elasticsearch have an index template for metricbeat-* when dashboards are loaded. I see the created indices and the mapping but if I call _templates on rest-api I do not get a template. If logstash managing templates i receive the template for logstash-*

[chris060986 (Christoph Birk)]

We figured it out. The problem is the missing index-templates for the beats. Thanks for the hints.

Filebeat Template Documentation describes to first upload the index templates to elasticsearch if using logstash in your pipeline between any beats and elasticsearch.

Solved!