Output.kafka filebeat (possible to add field ?)

Elastic Stack Beats
1

Hi,
I would like to know if it is possible to add a field in the kafka output
filebeat.yml:

output.kafka:
  hosts: ["127.0.0.1:9092"]
  topic: '%{[fields.kafka_topic]}'
  codec.format:
    string: '%{[message]}'
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000
  close_inactive: 50m
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

So i want to add field => "path" i found a solution but it's a bad solution i think i add this line in output.kafka

key: '%{[log][file][path]}'

It works but it means I change the value of "key" so it's a bad idea
logstash.conf

mutate{
    add_field => { "[path]" => "%{[@metadata][kafka][key]}"}
    }

Any help would be sincerely appreciate!
Thanks!

2

We have a list of filebeat processors that can modify the data before it is being sent to the output, you can add them after the two processors you already have:

And if you need to copy a value from an existing field:

There should be a few yml examples in those docs as well :slight_smile:

3

Thank you for the reply.
I have already tried these examples but didn't work i don't know where is the problem.
i tried this

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_fields:
      target: pathtest
      fields:
        name: '%{[log][file][path]}'

logstash.conf

mutate{
    add_field => { "[pathtest]" => "%{[pathtest]}"}
    }

it gives me
"pathtest" => "%{[pathtest]}"
and i also tried this:

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - add_fields:
      target: '@metadata'
      fields:
        op_type: "%{[log][file][path]}"

logstash.conf

mutate{
    add_field => { "[path]" => "%{[@metadata]}"}
    }

It gives me
"path" => "{\"kafka\":{\"topic\":\"kafka-topic-test\",\"consumer_group\":\"logstash\",\"partition\":0,\"offset\":72,\"key\":\"null\",\"timestamp\":1720134475223}}"
there is no op_type in metadata... i don't know if i did something wrong ?

Any help would be sincerely appreciate!
Thanks!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.