Metricbeat issue: failed to parse: Limit of total fields [10000] has been exceeded while adding new fields

Elastic Stack Beats
1

We use the AWS module to push SQS and RDS metrics to Metricbeat.
Everything is setup with ECK on Kubernetes and Helm.

Recently, probably after updating to 8.15.2, these metrics weren't being pushed anymore. The metricbeat containers showed the following error: [1:709] failed to parse: Limit of total fields [10000] has been exceeded while adding new fields [1].
We never set any limit ourselves, so I guess the limit of 10000 is default. Also we use the default index mapping.

When counting all the fields, we could see there were only 6678 fields. So our first thought was there was a different issue, since the limit of 10000 wasn't reached at all. After hours of debugging, as a last resort we set the limit higher and to our surprise: everything worked again. Counting the fields, there were now 10196 fields! A bit more than the old limit of 10000. Upon closer inspection, the only difference was a new property "type: object" on a lot of fields.

So, for now everything is working again but I would like to know if this is default behaviour, and should we have expected this issue?
Does our new limit automatically migrate over to the new index mapping if we update Metricbeat?

Also, we see a lot of fields we don't use in Metricbeat. From Airflow, GCP, Istio to Zookeeper. Is there any possibility to disable these fields via the Helm configuration? The smaller the amount of fields, the better I think.

2

Hello!

i faced this problem too, but couldn't find a better solution than just deleting some mapping in metricbeat Index Template.
go to Stack Management--->Index Management--->Index Templates, find your template version, most likely it will be a "metricbeat-8.15.2" in your case. edit it, go to mappings tab and randomly delete some mapping for platforms you won't use any further. then depending on your metricbeat lifecycle policy new indices will be created with new settings and all errors will gone or you can manually delete metricbeat datastream (in case you have one) to fasten this process. BUT YOU WILL LOSE all .ds-metricbeat indices in that case, so be very carefull in making that decision

1 Like
3

Thank you for your solution! Today after updating to 8.16.0 we ran into the same issue again. The cause was the new index template for 8.16.0 being used, which defaults back to the original settings.

So it looks like we need to do this manually after each update.

4

Add me to the "me too" list. I'm in the process of upgrading my cluster from 7.17.18 to 8.16.1. Started with trying to upgrade all the Beats deployments. Had no issues with Filebeat, Packetbeat, or Winlogbeat.

I updated Metricbeat to 8.16.1, ran metricbeat setup --index-management --pipelines, confirmed the index template, lifecycle policy, and pipelines were installed, and tried to start the service. Now I continually get the error message Limit of total fields [10000] has been exceeded while adding new fields [18]

I started with a clean metricbeat.yml file and added the settings needed for the output:elasticsearch section (hosts, username & password). The only modules I have enabled for beats-xpack and elasticsearch-xpack

I have this running on an Ubuntu 20.04 server using the package installers.

5

It looks like this has been addressed in the 8.17.0 update. The metricbeat index template is now mapped out of the box to 12500 instead of 10000 total fields:

    "mapping": {
      "total_fields": {
        "limit": "12500"
      }
    },

I can't seem to find something about it in the changelogs though.

6

I think it is already there. you might just have to change that number.
I can see this in my template.

"mapping": {
  "total_fields": {
    "limit": "10000"
  }
},