Filebeat "dynamic" field rename

[Stanislav_Lapshansky]

I want to rename fields in filebeat according to variable value. I've tried this:

filebeat.inputs:
- type:  docker
#...
  processors:
    - rename:
        fields:
          - from: "json"
            to: "%{['docker.container.name']}"
        ignore_missing: true
        fail_on_error: true

Without any success. Can I use some variable value in "to" parameter?

[jsoriano (Jaime Soriano)]

Hey @Stanislav_Lapshansky, welcome to discuss

I think this is not possible. Also take into account that given the nature of the docker.container.name, a behaviour like the one you expect would lead to fields explosion problems. You could end up with as many field names in your indexes as containers you have, and this can be problematic.

What are you expecting to achieve with this renaming?

[Stanislav_Lapshansky]

Dear Jaime, I'll try to explain

I have few docker containers. They send JSON logs to ElasticSearch via FileBeat. Some containers logs have fields with identical names but different types. So Elastic refuse indexing that log entries.

I can't fix mess with types of fields on dockers side. But I can control names of containers.

Now I solved the problem by creating separate index for each container name:

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  indices:
          - index: "filebeat-%{[beat.version]}-%{+yyyy.MM.dd}-%{[docker.container.name]}"
            when.has_fields: ['docker.container.name']

I think it is weak solution, because of too many indices in ElasticSearch. So I would to change fields names in FileBeat aim to make it unique via rename (fieldA -> dockername.fieldA for example)

[jsoriano (Jaime Soriano)]

@Stanislav_Lapshansky

I have few docker containers. They send JSON logs to Elasticsearch via FileBeat. Some containers logs have fields with identical names but different types. So Elastic refuse indexing that log entries.

What fields produce these conflicts? Are they specific of your applications?

Maybe another option is to provide an specific mapping for these fields, so they don't produce conflicts. This can be done with the setup.template.append_fields option.

@Stanislav_Lapshansky

I think it is weak solution, because of too many indices in Elasticsearch. So I would to change fields names in FileBeat aim to make it unique via rename (fieldA -> dockername.fieldA for example)

Agree, but take into account that having too many fields in the same index can also be problematic.

Another option to do custom renames could be to use the script processor, that allows to define any transformation using javascript or painless script.