Metricbeat issue: failed to parse: Limit of total fields [10000] has been exceeded while adding new fields

Elastic Stack Beats
1

We use the AWS module to push SQS and RDS metrics to Metricbeat.
Everything is setup with ECK on Kubernetes and Helm.

Recently, probably after updating to 8.15.2, these metrics weren't being pushed anymore. The metricbeat containers showed the following error: [1:709] failed to parse: Limit of total fields [10000] has been exceeded while adding new fields [1].
We never set any limit ourselves, so I guess the limit of 10000 is default. Also we use the default index mapping.

When counting all the fields, we could see there were only 6678 fields. So our first thought was there was a different issue, since the limit of 10000 wasn't reached at all. After hours of debugging, as a last resort we set the limit higher and to our surprise: everything worked again. Counting the fields, there were now 10196 fields! A bit more than the old limit of 10000. Upon closer inspection, the only difference was a new property "type: object" on a lot of fields.

So, for now everything is working again but I would like to know if this is default behaviour, and should we have expected this issue?
Does our new limit automatically migrate over to the new index mapping if we update Metricbeat?

Also, we see a lot of fields we don't use in Metricbeat. From Airflow, GCP, Istio to Zookeeper. Is there any possibility to disable these fields via the Helm configuration? The smaller the amount of fields, the better I think.

2

Hello!

i faced this problem too, but couldn't find a better solution than just deleting some mapping in metricbeat Index Template.
go to Stack Management--->Index Management--->Index Templates, find your template version, most likely it will be a "metricbeat-8.15.2" in your case. edit it, go to mappings tab and randomly delete some mapping for platforms you won't use any further. then depending on your metricbeat lifecycle policy new indices will be created with new settings and all errors will gone or you can manually delete metricbeat datastream (in case you have one) to fasten this process. BUT YOU WILL LOSE all .ds-metricbeat indices in that case, so be very carefull in making that decision

1 Like
3

Thank you for your solution! Today after updating to 8.16.0 we ran into the same issue again. The cause was the new index template for 8.16.0 being used, which defaults back to the original settings.

So it looks like we need to do this manually after each update.