Metricbeat 7.12.1 mapping error (cannot be changed from type)

[Flavio_Pompermaier]

Hi to all,
similarly to Metricbeat mapper error in 7.9.2 I have a problem when sending data from metricbeat to Logstash/Elasticsearch (7.12.1).
I have enabled 3 modules: system, elasticsearch-xpack and logstash-xpack.
Almost everything is ok but some document is not indexed due to some mapping problem.

There are 2 fields causing such an error:

• index_recovery.shards.index.size.reused_in_bytes (float -> long)
• stack_stats.xpack.ilm.policy_stats.phases.delete.min_age (long -> float)

Here the logged errors on Logstash:

[WARN ] [[metricbeat]>worker0] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"metricbeat-7.12.1", :routing=>nil}, #<LogStash::Event:0x6103cd9c>], :response=>{"index"=>{"_index"=>"metricbeat-7.12.1-2021.05.10-000001", "_type"=>"_doc", "_id"=>"ZAaUVXkB1TVUNgZeI7Q2", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [index_recovery.shards.index.size.reused_in_bytes] cannot be changed from type [float] to [long]"}}}}
[WARN ] [[metricbeat]>worker0] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"metricbeat-7.12.1", :routing=>nil}, #<LogStash::Event:0x4ff287ff>], :response=>{"index"=>{"_index"=>"metricbeat-7.12.1-2021.05.10-000001", "_type"=>"_doc", "_id"=>"jwaUVXkB1TVUNgZeI7Q2", "status"=>400, "error"=>{"type"=>"illegal_argument_exception", "reason"=>"mapper [stack_stats.xpack.ilm.policy_stats.phases.delete.min_age] cannot be changed from type [long] to [float]"}}}}

[Flavio_Pompermaier]

Actually the problem was caused by the fact that when using metricbeat to monitor the cluster, the elasticsearch output should be properly handled (but this is not documented at all in the documentation...). The hint to solve my issue comes from Elastic Stack monitoring with Metricbeat via Logstash or Kafka | Elastic Blog.

Summarizing this is what I had to add to my conf file:

filter{
  # Boilerplate for compatibility across Beats versions
  mutate {
     rename => { "[@metadata][id]" => "[@metadata][_id]" }
  }
}

output {
  if [@metadata][index] =~ /^.monitoring-*/ {
    # route stack monitoring data to monitoring Elasticsearch cluster
    if [@metadata][_id] {
      elasticsearch {
        index => "%{[@metadata][index]}-%{+YYYY.MM.dd}"
        document_id => "%{[@metadata][_id]}"
        hosts => ["https://es-host:9200"]
        ssl => true
        cacert => '/usr/share/elasticsearch/config/certificates/ca/ca.crt'
        user => logstash_internal
        password => 'xxx'
      }
    } else {
      elasticsearch{
        index => "%{[@metadata][index]}-%{+YYYY.MM.dd}"
        hosts => ["https://es-host:9200"]
        ssl => true
        cacert => '/usr/share/elasticsearch/config/certificates/ca/ca.crt'
        user => logstash_internal
        password => 'xxx'
      }
    }
  } else {
    # route non-stack monitoring data
    if [@metadata][pipeline] {
       elasticsearch {
          hosts => 'https://es-host:9200'
          ssl => true
          cacert => '/usr/share/elasticsearch/config/certificates/ca/ca.crt'
          user => logstash_internal
          password => 'xxx'
          manage_template => false
          index => "%{[@metadata][beat]}-%{[@metadata][version]}"
          ilm_enabled => false
          pipeline => "%{[@metadata][pipeline]}"
       }
     } else {
       elasticsearch {
         hosts => 'https://es-host:9200'
         ssl => true
         cacert => '/usr/share/elasticsearch/config/certificates/ca/ca.crt'
         user => logstash_internal
         password => 'xxx'
         manage_template => false
         index => "%{[@metadata][beat]}-%{[@metadata][version]}"
         ilm_enabled => false
       }
     }
  }
}

I hope this could help someone else..