Filebeats creating the incorrect datatype in index

Elastic Stack Beats
1

I find this perplexing but it seems as though filebeats has decided to take an input that is always a text object with no numerals in it and index it as a date object. As a result the filebeats data is rejected by ES because of date formatting issues.

How can someone make sure that the index is being properly created by Filebeats?

2

Hi @mbradsher and welcome,

Could you give more details about your problem? What version of filebeat are you using? What field is unexpectedly being converted to date?

3

The version we are using 7.2.1. It takes two log types from within an ECS container. The one that is failing is posted here:

type: log
    enabled: true
    paths:
      - /dir1/dir2/dir3/dir4/%{var1}/%{var2}/%{var3}/*.log
      - /dir1/dir2/dir3/dir4/*/*/*/*.log
    multiline.pattern: ^\[
    multiline.match: after
    multiline.negate: true
    tags: ["index1"]
    fields:
      app_name: "myindex_1"
    processors:
      - dissect:
          tokenizer: "/%{}/%{}/%{}/%{}/%{var1}/%{var2}/%{var3}/%{var4}.log"
          field: "log.file.path"
          target_prefix: "myindex_"

Essentially var3 is a database table name but the index in elasticsearch is being created with a 'date' type. Then when a log file is created we get an illegal_argument_exception.

{"type":"illegal_argument_exception","reason":"failed to parse date field [<table name>] with format [strict_date_optional_time||epoch_millis]","caused_by":{"type":"date_time_parse_exception","reason":"date_time_parse_exception: Failed to parse with all enclosed parsers"}}}

Is this a case where we want to create an Elasticsearch index template or is there a simpler solution? I'm still very new to this tech stack.

4

But what is the real name of var3 and <table name>? I am asking to check if there is already a predefined field with this name formatted as date. Send it to me to a private message if you want to avoid making it public.

If there is a predefined field, you may consider to use another field that is not currently used.

If you want to completely ignore the predefined mappings and define your own, you can try to use the setup.template.fields option. You can find examples of complete fields.yml files by downloading the beats code and running make fields from a beat directory.

5

Thank you for your help. The real value of var3 is 'stepid' and the issue only happens intermittently so I do not expect this is from a name collision. Instead I took your suggestion to be explicit about the mapping by using the experimental setup.template.append_fields property.

I've run into new problems but they are expressed on a new forum here.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.