# ES 2.3 -\> 5.x metricbeat index field limit **URL:** \ **Category:** Beats\ **Tags:** metricbeat\ **Created:** [November 22, 2016, 8:01am UTC](https://discuss.elastic.co/t/es-2-3-5-x-metricbeat-index-field-limit/66821 "2016-11-22T08:01:28Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![gregoryo](https://avatars.discourse-cdn.com/v4/letter/g/34f0e0/32.png) [@gregoryo](https://discuss.elastic.co/u/gregoryo)\ **Post date:** [November 22, 2016, 8:01am UTC](https://discuss.elastic.co/t/es-2-3-5-x-metricbeat-index-field-limit/66821/1 "2016-11-22T08:01:28Z")
I'm migrating ES 2.3 -\> 5.x. Using the [migration plugin](https://github.com/elastic/elasticsearch-migration/tree/2.x) I get a Cluster Checkup warning for my logstash-metricbeat-\* indices (all from metricbeat): "New indices may not have more than 1000 fields. This index has 1719." Has metricbeat been fixed to avoid this problem? Is there something else I can do about it? Thanks, Greg. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [November 22, 2016, 8:55am UTC](https://discuss.elastic.co/t/es-2-3-5-x-metricbeat-index-field-limit/66821/2 "2016-11-22T08:55:14Z")
We are aware that we will face this issue in the near future but I'm kind of surprised that you already hit this issue now as the total number of fields in metricbeat should be still \< 1000. Do you have anything else in this index? Can you share your config file? As a workaround you can adjust to the `index.mapping.total_fields.limit`: [https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#mapping-limit-settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#mapping-limit-settings) ---
**Author:** ![gregoryo](https://avatars.discourse-cdn.com/v4/letter/g/34f0e0/32.png) [@gregoryo](https://discuss.elastic.co/u/gregoryo)\ **Post date:** [November 22, 2016, 1:16pm UTC](https://discuss.elastic.co/t/es-2-3-5-x-metricbeat-index-field-limit/66821/3 "2016-11-22T13:16:28Z")
This is my typical metricbeat.yml, used on most clients: ``` metricbeat.modules: - module: system metricsets: - cpu - filesystem - memory period: 60s output.logstash: hosts: ['logstash-server:5043'] index: metricbeat tls.certificate_authorities: ["/usr/local/share/certs/managed/ca/logstash-ca.pem"] tls.certificate: "/usr/local/share/certs/managed/raw/this-host.pem" tls.certificate_key: "/usr/local/share/certs/managed/raw/this-host.key" ``` The only difference is on one client I have seven metricsets (cpu core diskio filesystem fsstat memory network) enabled. yes, I suppose I could increase the limit to 2000 to get the upgrade done. That's a good way forward for now. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [November 22, 2016, 1:31pm UTC](https://discuss.elastic.co/t/es-2-3-5-x-metricbeat-index-field-limit/66821/4 "2016-11-22T13:31:11Z")
Few additional questions to figure out where all the fields come from on your side: - Are you doing any transformations on the logstash side? - Do you use daily indices? - Did you load the template into elasticsearch? It is automatically loaded if you send data to elasticsearch. - Did you use "beta" versions of metricbeat on the same indices? ---
**Author:** ![gregoryo](https://avatars.discourse-cdn.com/v4/letter/g/34f0e0/32.png) [@gregoryo](https://discuss.elastic.co/u/gregoryo)\ **Post date:** [November 22, 2016, 2:29pm UTC](https://discuss.elastic.co/t/es-2-3-5-x-metricbeat-index-field-limit/66821/5 "2016-11-22T14:29:32Z")
> [@](#): > > Are you doing any transformations on the logstash side? No filters, just this in output: ``` output { if [type] == 'metricsets' { elasticsearch { index => "logstash-metricbeat-%{+xxxx.ww}" document_type => "%{[@metadata][type]}" ``` > [@](#): > > Do you use daily indices? Weekly, as you can see above. > [@](#): > > Did you load the template into elasticsearch? It is automatically loaded if you send data to elasticsearch. In short, yes I did, and it's described more [here](https://discuss.elastic.co/t/metricbeat-ls-es-template-error-solved). > [@](#): > > Did you use "beta" versions of metricbeat on the same indices? Since 2 November I've been using a metricbeat that I [built myself](https://github.com/elastic/beats/issues/974#issuecomment-229604457), at whatever versions of things at that time. I have nothing else in those indices (logstash-metricbeat-2016.{44-47}), and no earlier logstash-metricbeat-\* indices. I wonder if it's time to build the binary again from current sources. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [November 23, 2016, 10:53am UTC](https://discuss.elastic.co/t/es-2-3-5-x-metricbeat-index-field-limit/66821/6 "2016-11-23T10:53:11Z")
Thanks for all the details. My current assumption is that one weekly indices has \> 1000 fields which is probably the one around november 2 when you first started with Logstash and applying templates. I expect all current indices don't have that many fields. I would recommend you to load the new template for 5.x so it will apply on the new indices. ---
**Author:** ![gregoryo](https://avatars.discourse-cdn.com/v4/letter/g/34f0e0/32.png) [@gregoryo](https://discuss.elastic.co/u/gregoryo)\ **Post date:** [November 24, 2016, 2:10am UTC](https://discuss.elastic.co/t/es-2-3-5-x-metricbeat-index-field-limit/66821/7 "2016-11-24T02:10:10Z")
Thank you for the help so far. > [@ruflin](#): > > My current assumption is that one weekly indices has \> 1000 The migration helper tells me that: - .44 has 1721 fields - .45-47 have 1719 fields each I will increase the limit to 2000, upgrade, apply the current version of the template, and see what happens from there. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [November 25, 2016, 10:02am UTC](https://discuss.elastic.co/t/es-2-3-5-x-metricbeat-index-field-limit/66821/8 "2016-11-25T10:02:08Z")
It would be interesting to see what your current mapping is to see if there are some fields inside we are not aware of: `GET /metricbeat-*/_mapping` This will be quite long, so best put it into a gist. ---
**Author:** ![gregoryo](https://avatars.discourse-cdn.com/v4/letter/g/34f0e0/32.png) [@gregoryo](https://discuss.elastic.co/u/gregoryo)\ **Post date:** [November 28, 2016, 1:16am UTC](https://discuss.elastic.co/t/es-2-3-5-x-metricbeat-index-field-limit/66821/9 "2016-11-28T01:16:37Z")
Okay, here it is: > ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [November 28, 2016, 10:09am UTC](https://discuss.elastic.co/t/es-2-3-5-x-metricbeat-index-field-limit/66821/10 "2016-11-28T10:09:32Z")
I had a closer look at the mapping `44` an it seems it contains the complete mapping twice but a second time under `metricset` namespace: ```auto "metricsets": { "_meta": { "version": "6.0.0-alpha1" }, "_all": { "norms": false }, "dynamic_templates": [ { "strings_as_keyword": { "match_mapping_type": "string", "mapping": { "ignore_above": 1024, "type": "keyword" } } } ], "properties": { "@timestamp": { "type": "date" }, "apache": { "properties": { "status": { "properties": { "bytes_per_request": { "type": "scaled_float", "scaling_factor": 1000 }, "bytes_per_sec": { ``` I'm not sure how this could have happened? How did you generate / load the template? ---
**Author:** ![gregoryo](https://avatars.discourse-cdn.com/v4/letter/g/34f0e0/32.png) [@gregoryo](https://discuss.elastic.co/u/gregoryo)\ **Post date:** [December 5, 2016, 4:51am UTC](https://discuss.elastic.co/t/es-2-3-5-x-metricbeat-index-field-limit/66821/11 "2016-12-05T04:51:43Z")
Sorry for the slow reply. I've updated the relevant [topic](https://discuss.elastic.co/t/metricbeat-ls-es-template-error-solved/64671) with some detail (that is, I manually downloaded and installed the template), and now wonder about the version issue. I have successfully upgraded 2.3 -\> 5.0, so should I now manually replace the template with a 5.0 one? ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [December 6, 2016, 8:44am UTC](https://discuss.elastic.co/t/es-2-3-5-x-metricbeat-index-field-limit/66821/12 "2016-12-06T08:44:55Z")
In case you didn't make any adjustments to the template, I strongly recommend to overwrite it. You don't have to do it manually but can set `overwrite` to true in the metricbeat config file. As long as you don't modify the template yourself, having overwrite to true can become an issue if you have multiple metricbeat instances with different versions at the same time. But overwriting only happens once on startup. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 3, 2017, 8:45am UTC](https://discuss.elastic.co/t/es-2-3-5-x-metricbeat-index-field-limit/66821/13 "2017-01-03T08:45:13Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.