Metricbeat 7.17.6 - Cannot index event

Elastic Stack Beats
1

Following our cluster upgrade to Elasticsearch 7.17.6, we had to migrate the monitoring stack using Metricbeat.
In Kibana, monitoring is not working, hosts are marked as unmonitored even though Metricbeat is on all hosts and running.

So I looked at the Metricbeat logs, and I came across these errors that keep recurring:

2022-10-07T09:04:35.103Z        WARN        [elasticsearch]        elasticsearch/client.go:414        Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.October, 7, 9, 4, 33, 493115287, time.Local), Meta:{"index":".monitoring-es-7-mb"}, Fields:{..........}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"illegal_argument_exception","reason":"mapper [index_recovery.shards.index.size.recovered_in_bytes] cannot be changed from type [float] to [long]"}, dropping event!
2022-10-07T09:04:35.185Z        WARN        [elasticsearch]        elasticsearch/client.go:414        Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.October, 7, 9, 4, 33, 516565264, time.Local), Meta:{"index":".monitoring-es-7-mb"}, Fields:{..........}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_argument_exception","reason":"Limit of total fields [1000] has been exceeded while adding new fields [631]"}}, dropping event!

My knowledge of Metricbeat/Elasticsearch being very basic (reading documentation), I don't know what to do.
Is it a bad configuration ? A bug ?

For information, here is the Metricbeat configuration:

metricbeat:
  config:
    modules:
      path: /etc/metricbeat/modules.d/*.yml
      reload:
        enabled: false
name: svc-els-master-001
output:
  elasticsearch:
    hosts:
    - svc-els-data-001:9200
    password: xXxXxXxXxXx
    username: xXxXxXxXxXx
path:
  config: /etc/metricbeat
  data: /var/lib/metricbeat
  home: /usr/share/metricbeat
  logs: /var/log/metricbeat
processors:
- add_host_metadata: null
- add_cloud_metadata: null
- add_docker_metadata: null
- add_kubernetes_metadata: null
setup:
  kibana: null
  template:
    settings:
      index:
        codec: best_compression
        number_of_shards: 1

Elasticsearch version: Version: 7.17.6, Build: default/deb/f65e9d338dc1d07b642e14a27f338990148ee5b6/2022-08-23T11:08:48.893373482Z, JVM: 18.0.2
Metricbeat version: metricbeat version 7.17.6 (amd64), libbeat 7.17.6 [121b2ab96f118e6999f0a4be8e98827a809337db built 2022-08-23 12:50:12 +0000 UTC]

Thank you for your help

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.