ES 2.3 -> 5.x metricbeat index field limit

Elastic Stack Beats
1

I'm migrating ES 2.3 -> 5.x. Using the migration plugin I get a Cluster Checkup warning for my logstash-metricbeat-* indices (all from metricbeat):

"New indices may not have more than 1000 fields. This index has 1719."

Has metricbeat been fixed to avoid this problem? Is there something else I can do about it?

Thanks,
Greg.

2

We are aware that we will face this issue in the near future but I'm kind of surprised that you already hit this issue now as the total number of fields in metricbeat should be still < 1000. Do you have anything else in this index? Can you share your config file?

As a workaround you can adjust to the index.mapping.total_fields.limit: https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#mapping-limit-settings

3

This is my typical metricbeat.yml, used on most clients:

metricbeat.modules:
- module: system
  metricsets:
    - cpu
    - filesystem
    - memory
  period: 60s
output.logstash:
  hosts: ['logstash-server:5043']
  index: metricbeat
  tls.certificate_authorities: ["/usr/local/share/certs/managed/ca/logstash-ca.pem"]
  tls.certificate: "/usr/local/share/certs/managed/raw/this-host.pem"
  tls.certificate_key: "/usr/local/share/certs/managed/raw/this-host.key"

The only difference is on one client I have seven metricsets (cpu core diskio filesystem fsstat memory network) enabled.

yes, I suppose I could increase the limit to 2000 to get the upgrade done. That's a good way forward for now.

4

Few additional questions to figure out where all the fields come from on your side:

  • Are you doing any transformations on the logstash side?
  • Do you use daily indices?
  • Did you load the template into elasticsearch? It is automatically loaded if you send data to elasticsearch.
  • Did you use "beta" versions of metricbeat on the same indices?
5

No filters, just this in output:

output {
  if [type] == 'metricsets' {
    elasticsearch {
      index => "logstash-metricbeat-%{+xxxx.ww}"
      document_type => "%{[@metadata][type]}"

Weekly, as you can see above.

In short, yes I did, and it's described more here.

Since 2 November I've been using a metricbeat that I built myself, at whatever versions of things at that time. I have nothing else in those indices (logstash-metricbeat-2016.{44-47}), and no earlier logstash-metricbeat-* indices. I wonder if it's time to build the binary again from current sources.

6

Thanks for all the details. My current assumption is that one weekly indices has > 1000 fields which is probably the one around november 2 when you first started with Logstash and applying templates. I expect all current indices don't have that many fields.

I would recommend you to load the new template for 5.x so it will apply on the new indices.

7

Thank you for the help so far.

The migration helper tells me that:

  • .44 has 1721 fields
  • .45-47 have 1719 fields each

I will increase the limit to 2000, upgrade, apply the current version of the template, and see what happens from there.

8

It would be interesting to see what your current mapping is to see if there are some fields inside we are not aware of: GET /metricbeat-*/_mapping This will be quite long, so best put it into a gist.

9

Okay, here it is:

10

I had a closer look at the mapping 44 an it seems it contains the complete mapping twice but a second time under metricset namespace:

      "metricsets": {
        "_meta": {
          "version": "6.0.0-alpha1"
        },
        "_all": {
          "norms": false
        },
        "dynamic_templates": [
          {
            "strings_as_keyword": {
              "match_mapping_type": "string",
              "mapping": {
                "ignore_above": 1024,
                "type": "keyword"
              }
            }
          }
        ],
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "apache": {
            "properties": {
              "status": {
                "properties": {
                  "bytes_per_request": {
                    "type": "scaled_float",
                    "scaling_factor": 1000
                  },
                  "bytes_per_sec": {

I'm not sure how this could have happened? How did you generate / load the template?

11

Sorry for the slow reply. I've updated the relevant topic with some detail (that is, I manually downloaded and installed the template), and now wonder about the version issue. I have successfully upgraded 2.3 -> 5.0, so should I now manually replace the template with a 5.0 one?

12

In case you didn't make any adjustments to the template, I strongly recommend to overwrite it. You don't have to do it manually but can set overwrite to true in the metricbeat config file. As long as you don't modify the template yourself, having overwrite to true can become an issue if you have multiple metricbeat instances with different versions at the same time. But overwriting only happens once on startup.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.