Filebeat "dynamic" field rename

I want to rename fields in filebeat according to variable value. I've tried this:

- type:  docker
    - rename:
          - from: "json"
            to: "%{['']}"
        ignore_missing: true
        fail_on_error: true

Without any success. Can I use some variable value in "to" parameter?

Hey @Stanislav_Lapshansky, welcome to discuss :slightly_smiling_face:

I think this is not possible. Also take into account that given the nature of the, a behaviour like the one you expect would lead to fields explosion problems. You could end up with as many field names in your indexes as containers you have, and this can be problematic.

What are you expecting to achieve with this renaming?

Dear Jaime, I'll try to explain

I have few docker containers. They send JSON logs to ElasticSearch via FileBeat. Some containers logs have fields with identical names but different types. So Elastic refuse indexing that log entries.

I can't fix mess with types of fields on dockers side. But I can control names of containers.

Now I solved the problem by creating separate index for each container name:

  hosts: ["elasticsearch:9200"]
          - index: "filebeat-%{[beat.version]}-%{+yyyy.MM.dd}-%{[]}"
            when.has_fields: ['']

I think it is weak solution, because of too many indices in ElasticSearch. So I would to change fields names in FileBeat aim to make it unique via rename (fieldA -> dockername.fieldA for example)

What fields produce these conflicts? Are they specific of your applications?

Maybe another option is to provide an specific mapping for these fields, so they don't produce conflicts. This can be done with the setup.template.append_fields option.

Agree, but take into account that having too many fields in the same index can also be problematic.

Another option to do custom renames could be to use the script processor, that allows to define any transformation using javascript or painless script.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.