8.3.1 -> 8.3.2 illegal_argument_exception geo.country_iso_code

[mayer (Hans)]

Dear All,

I upgraded from 8.3.1 to 8.3.2 and now I get an error when I access a geo map in kibana with the message: 1 of 2 shards failed

the detailed information is

        "index": "filebeat-8.3.2",
        "reason": {
          "type": "illegal_argument_exception",
          "reason": "Text fields are not optimised for operations that require per-document field data like aggregations and sorting, so these operations are disabled by default. Please use a keyword field instead. Alternatively, set fielddata=true on [bind9.geo.country_iso_code] in order to load field data by uninverting the inverted index. Note that this can use significant memory."

In version 8.3.1 I created a new module in filebeat with name "bind9".
In manifest I defined

requires.processors:
- name: geoip
  plugin: ingest-geoip

When I define mappings I have several fields but nothing special for geo.
In the pipeline.yml there is:

- geoip:
    field: bind9_remoteip
    target_field: bind9.geo
    ignore_missing: true

Looking for mapping I see

GET /filebeat-*/_mapping/field/bind9.geo.country_iso_code

{
  "filebeat-8.3.2": {
    "mappings": {
      "bind9.geo.country_iso_code": {
        "full_name": "bind9.geo.country_iso_code",
        "mapping": {
          "country_iso_code": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      }
    }
  },
  ".ds-filebeat-8.3.1-2022.07.13-000001": {
    "mappings": {
      "bind9.geo.country_iso_code": {
        "full_name": "bind9.geo.country_iso_code",
        "mapping": {
          "country_iso_code": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      }
    }
  }
}

Any idea how to add "fielddata=true" or to change from "text" to "keyword" ?

Any help is welcome.

Kind regards
Hans

[BenB196]

Any chance the index template for 8.3.2 wasn't properly loaded before you started ingesting data from the 8.3.2 version? These mappings:

"mapping": {
  "country_iso_code": {
    "type": "text",
    "fields": {
      "keyword": {
        "type": "keyword",
        "ignore_above": 256
      }
    }
  }
}

Look like the default Logstash/Elasticsearch mappings, rather than mappings provided by Filebeat (note: I didn't check the source code of Filebeat 8.3.2, so it is possible that a mapping bug made its way through)

[mayer (Hans)]

Hi, before I run an upgrade I stop complete ELK stack. Also remote filebeat. Do the upgrade, start Elasticsearch and kibana. Manually I post a single doc to load the index because I have to define a lot of fields before I start all filebeats ( the local and remote ) Definitely there was a lot of time before data from filebeat was coming.
I am wondering there is a datastream only for version 8.3.1 but not for 8.3.2 ( for all beats ). But there are index templates for 8.3.1 and 8.3.2
Is this normal ? Can I test something ?

// Hans

[stephenb (Stephen Brown)]

Are you running

filebeat setup -e

For each version before you start the first filebeat shipping logs? Or index your test doc?

If not that is probably contributing to your issue.

[BenB196]

There are different index templates for 8.3.1 and 8.3.2, you need to load the index templates & create the matching data stream (this can be done with via the normal Filebeat setup proceedure after an upgrade), before creating docs. If you just create a doc without doing the above, it will just create a default index with the default mappings, which isn't what you want.

You can check this by looking at the index template loaded into the Elasticsearch instance to see if there is an 8.3.2 index template. Then you can check to see if there is a properly created data stream for Filebeat 8.3.2 (in 8.x beats switched from using a standard index to using a data stream)

If the above 2 things aren't there, then you probably need to go back and run the Filebeat setup for 8.3.2 (you might need to delete the existing 8.3.2 index if one exists as you can't have a data stream with the same name as an index; Warning: Deleting an index will cause the data on it to be deleted, be sure to have a backup of it, if you don't want to lose the data.)

[mayer (Hans)]

Dear Ben, dear Stephen,
many thanks for your support. It's working now. Great.
What I have done:

• stopped all filebeats
• for filebeat I deleted all indices, data streams and index templates ( yes, there was also an index for filebeat )
• run "filebeat setup -e"
• checked if index template is there
• run my script to add some fields
• started all filebeats
  And I am happy it's working now.

Now I have correct settings

GET /filebeat-*/_mapping/field/bind9.geo.country_iso_code

{
  ".ds-filebeat-8.3.2-2022.07.25-000001": {
    "mappings": {
      "bind9.geo.country_iso_code": {
        "full_name": "bind9.geo.country_iso_code",
        "mapping": {
          "country_iso_code": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      }
    }
  },
  ".ds-filebeat-8.3.1-2022.07.13-000001": {
    "mappings": {
      "bind9.geo.country_iso_code": {
        "full_name": "bind9.geo.country_iso_code",
        "mapping": {
          "country_iso_code": {
            "type": "keyword",
            "ignore_above": 1024
          }
        }
      }
    }
  }
}

The DS for 8.3.1 is there because I made a restore ( so I believe ?? )

// Hans