Force new field to type "keyword" or "text"

[yquirion]

Hello,

I'm currently struggling with an annoying problem who lead to many lost logs into my Elastic cluster.

The problem happen when a field that hasn't been defined into the default filebeat template is created. When the filebeat is sending a data with a new field, the field will be created into the index. But the type of this field may not always be the same.

For example, if the value contain a number at first, the field will be created as "long". If the field as a string as its value at first, it field will be "keyword".

This is causing lots of problem. Here is a very nice error I'm getting:

response: {"index"=>{"_index"=>"filebeat-8.6.2-o365-2023.12.19", "_id"=>"4ENEgowBcZxL1t679ex9", "status"=>400, "error"=>{"type"=>"document_parsing_exception", "reason"=>"[1:1064] failed to parse field [o365.audit.Platform] of type [long] in document with id '4ENEgowBcZxL1t679ex9'. Preview of field's value: 'OfficeCollaborationService'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"For input string: \"OfficeCollaborationService\""}}}}

Because the first value for the field o365.audit.Platform was a number, the field has been created with a type of long. Why the first value had a number in it? I can't tell that, this comes from Office365. But I know that value must be a string. When another record comes with a sting for that field, it will failed to parse the data and it will be lost. In the example above, we can see the value it tries to parse on field 365.audit.Platform is OfficeCollaborationService. This one will failed because the destination field has a type of long.

Since a number can fit into a test or keyword field, is there a way to tell Elastic to create any fields, that isn't part of the index template that comes with filebeat, usunbg a type of text or keyword?

From this index, I currently have 5 conflicts, all related to keyword,long type.

If there is a way to solve this situation, I will appreciate it!

If you need more information on this issue, please let me know.

Thank you all and best regards,
Yanick

[stephenb (Stephen Brown)]

Hi @yquirion

Yup that is an issue.

Perhaps Look at this.. Dynamic Templates I think this will help solve your issue

Dynamic templates | Elasticsearch Guide [8.11] | Elastic

You could try something like (not saying this is exactly correct)

You can add several of these to the mapping to control the behavior.

It pretty powerful

{
  "mappings": {
    "dynamic_templates": [
      {
        "o365_audit": {
          "path_match": "o365.audit.*"
          "mapping": {
            "type": "keyword"
          }
        }
      }
    ]
  }
}

[yquirion]

@stephenb

"type": "keyword"

Hi @stephenb,

Thanks again for your reply.

I'm already using dynamic templates and yes they are working well. I was so focused on the field in error that I didn't think about adding what you just sent me

Here is what I already have:

"dynamic_templates": [
  {
    "labels": {
      "path_match": "labels.*",
      "mapping": {
        "type": "keyword"
      },
      "match_mapping_type": "string"
    }
  },
  {
    "container.labels": {
      "path_match": "container.labels.*",
      "mapping": {
        "type": "keyword"
      },
      "match_mapping_type": "string"
    }
  },
  {
    "fields": {
      "path_match": "fields.*",
      "mapping": {
        "type": "keyword"
      },
      "match_mapping_type": "string"
    }
  },
  {
    "docker.container.labels": {
      "path_match": "docker.container.labels.*",
      "mapping": {
        "type": "keyword"
      },
      "match_mapping_type": "string"
    }
  },
  {
    "kubernetes.labels.*": {
      "path_match": "kubernetes.labels.*",
      "mapping": {
        "type": "keyword"
      },
      "match_mapping_type": "*"
    }
  },
  {
    "kubernetes.annotations.*": {
      "path_match": "kubernetes.annotations.*",
      "mapping": {
        "type": "keyword"
      },
      "match_mapping_type": "*"
    }
  },
  {
    "kubernetes.selectors.*": {
      "path_match": "kubernetes.selectors.*",
      "mapping": {
        "type": "keyword"
      },
      "match_mapping_type": "*"
    }
  },
  {
    "docker.attrs": {
      "path_match": "docker.attrs.*",
      "mapping": {
        "type": "keyword"
      },
      "match_mapping_type": "string"
    }
  },
  {
    "azure.activitylogs.identity.claims.*": {
      "path_match": "azure.activitylogs.identity.claims.*",
      "mapping": {
        "type": "keyword"
      },
      "match_mapping_type": "*"
    }
  },
  {
    "kibana.log.meta": {
      "path_match": "kibana.log.meta.*",
      "mapping": {
        "type": "keyword"
      },
      "match_mapping_type": "string"
    }
  },
  {
    "ip_addr": {
      "path_match": "*.ip",
      "mapping": {
        "type": "ip"
      }
    }
  },
  {
    "ports": {
      "path_match": "*.port",
      "mapping": {
        "type": "float"
      }
    }
  },
  {
    "bytes": {
      "path_match": "*.bytes",
      "mapping": {
        "type": "float"
      }
    }
  },
  {
    "geo-point": {
      "path_match": "*.geo.location",
      "mapping": {
        "type": "geo_point"
      }
    }
  },
  {
    "event_original": {
      "path_match": "event.original",
      "mapping": {
        "norms": false,
        "type": "text"
      }
    }
  },
  {
    "json_exception": {
      "path_match": "json.exception",
      "mapping": {
        "norms": false,
        "type": "text"
      }
    }
  },
  {
    "strings_as_keyword": {
      "mapping": {
        "ignore_above": 1024,
        "type": "keyword"
      },
      "match_mapping_type": "string"
    }
  },
*** WILL ADD THIS HERE ***  
  {
    "o365_audit" : {
      "path_match" : "o365.audit.*"
      "mapping" : {
        "type" : "keyword"
      }
    }
  },
],

Will test it and let you know how it goes.

Thanks!
Yanick

[yquirion]

Hi @stephenb,

Sorry for the late reply, but just wanna letting you know that it worked using the dynamic template. I had other little issues, but was able to fix everything!

Thanks a lot for the reminder!

Regards,
Yanick