Metricbeat -> LS -> ES, template error [SOLVED]

Elastic Stack Logstash
1

I have been successfully using Filebeat -> LS -> ES for a few months. I now want to add Metricbeat as another source. I started ingesting data fine, but continued reading docs.
https://www.elastic.co/guide/en/beats/metricbeat/current/config-metricbeat-logstash.html tells me to load the index template manually, so did so (after stopping Metricbeat and deleting the index):

$ curl https://raw.githubusercontent.com/elastic/beats/f0b52e1926ef88dff0fa17f5341c16144a831c56/metricbeat/metricbeat.template.json > metricbeat.template.json
$ curl -XPUT 'http://localhost:9200/_template/metricbeat' -d@metricbeat.template.json
{"acknowledged":true}

At that point I start getting these errors in logstash.log:

"status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"Failed to parse mapping [_default_]: No handler for type [keyword] declared on field [hostname]", "caused_by"=>{"type"=>"mapper_parsing_exception", "reason"=>"No handler for type [keyword] declared on field [hostname]"}}}}, :level=>:warn}

I used this page was too:
https://www.elastic.co/guide/en/beats/metricbeat/current/logstash-output.html

metricbeat.yml snippet:

output.logstash:
  index: metricbeat

logstash.conf snippet:

output {
  if [type] == 'metricsets' {
    elasticsearch {
      index => "%{[@metadata][beat]}-%{+xxxx.ww}"
      document_type => "%{[@metadata][type]}"

Can anyone suggest to me what might be wrong here? I think I would ultimately like to use indices named logstash-metricbeat-year-week, but sticking with default metricbeat-year-week while getting past this error.

2

which version of elasticsearch are you using?

[edit]

metric beat provides 2 templates, one for elasticsearch 5.x (metricbeat.template.json) and another for 2.x (metricbeat.template-es2x.json)

3

Excellent point, thank you. I wondered whether perhaps that 'keyword' was a new thing in 5.x, but there were so many uses of it in the template that I was hoping for a shortcut to the faultfinding - which you've given me. I am still using 2.3, waiting for 5.x to become available in the FreeBSD ports system.

I used the -es2x version of the template with a 's/metricbeat-/logstash-metricbeat-/', and all appears well.

Cheers,
Greg.

1 Like
6

Some details for future reference: I found the templates here, imported the new version (5?) template, then after @jsvd's suggestion imported the version 2 template.

1 Like