Filebeat output kafka (processors)

Elastic Stack Beats
1

Hi,
I have a problem i think in "processors" , I use the output kafka so i created a topic and i send it to logstash so when i run it i just get the message when i want see path,tags,input type....
here my filebeat config:

output.kafka:
  hosts: ["127.0.0.1:9092"]
  topic: "kafka-topic"
  codec.format:
    string: '%{[message]}'
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

this one it gives me _id _index type @timestamp version message.

codec.json it gives me all data but i dont want it in format json

logstash.conf

input {
kafka {
      bootstrap_servers => "127.0.0.1:9092"
      topics => "kafka-topic"
    }
}
output{
elasticsearch {
      hosts => ["127.0.0.1:9200"]
      index => "kafka-%{+YYYY.MM.dd}"
  }
}

Should i add somthing or ... ?

Any help would be sincerely appreciate!
Thanks!

2

Same problem :confused: i get just 6 field i cant find @metadata when i use
codec.format:
string: '%{[message]}'

3

If you are using codec.format as a string, you will need to specify the fields you want to be shipped, like the example in the documentation.

The add_host_metadata will add host.* fields, and the add_cloud_metadata will add cloud.* fields.

So you should try to use:

codec.format:
  string: '%{[host]} %{[cloud]} %{[message]}'
1 Like
4

Thank you very much for the reply!
I tried this but the %{[cloud]} didn't work without any error...
So I tried

codec.format:
  string: '%{[host]} %{[message]}'

but it gives me host like json format under message, there is no possibility to add each one on field like filebeat output to Elasticsearch?

I mean like this maybe?

image
image927×335 16.6 KB

The most important fields that i want are log.file.path and tags if it's possible please!

Thanks!

5

Which cloud provider are you using? You can check if your provider is supported in the documentation.

The host is a json object, if you do not want to add it as a json you will need to add every field individually.

Like this:

codec.format:
  string: '%{[host][os][platform]} %{[host][os][version]} %{[message]}'
1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.