Following our cluster upgrade to Elasticsearch 7.17.6, we had to migrate the monitoring stack using Metricbeat.
In Kibana, monitoring is not working, hosts are marked as unmonitored even though Metricbeat is on all hosts and running.
So I looked at the Metricbeat logs, and I came across these errors that keep recurring:
2022-10-07T09:04:35.103Z WARN [elasticsearch] elasticsearch/client.go:414 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.October, 7, 9, 4, 33, 493115287, time.Local), Meta:{"index":".monitoring-es-7-mb"}, Fields:{..........}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"illegal_argument_exception","reason":"mapper [index_recovery.shards.index.size.recovered_in_bytes] cannot be changed from type [float] to [long]"}, dropping event!
2022-10-07T09:04:35.185Z WARN [elasticsearch] elasticsearch/client.go:414 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.October, 7, 9, 4, 33, 516565264, time.Local), Meta:{"index":".monitoring-es-7-mb"}, Fields:{..........}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_argument_exception","reason":"Limit of total fields [1000] has been exceeded while adding new fields [631]"}}, dropping event!
My knowledge of Metricbeat/Elasticsearch being very basic (reading documentation), I don't know what to do.
Is it a bad configuration ? A bug ?
For information, here is the Metricbeat configuration:
metricbeat:
config:
modules:
path: /etc/metricbeat/modules.d/*.yml
reload:
enabled: false
name: svc-els-master-001
output:
elasticsearch:
hosts:
- svc-els-data-001:9200
password: xXxXxXxXxXx
username: xXxXxXxXxXx
path:
config: /etc/metricbeat
data: /var/lib/metricbeat
home: /usr/share/metricbeat
logs: /var/log/metricbeat
processors:
- add_host_metadata: null
- add_cloud_metadata: null
- add_docker_metadata: null
- add_kubernetes_metadata: null
setup:
kibana: null
template:
settings:
index:
codec: best_compression
number_of_shards: 1
Elasticsearch version: Version: 7.17.6, Build: default/deb/f65e9d338dc1d07b642e14a27f338990148ee5b6/2022-08-23T11:08:48.893373482Z, JVM: 18.0.2
Metricbeat version: metricbeat version 7.17.6 (amd64), libbeat 7.17.6 [121b2ab96f118e6999f0a4be8e98827a809337db built 2022-08-23 12:50:12 +0000 UTC]
Thank you for your help