Filebeat output kafka (processors)

I have a problem i think in "processors" , I use the output kafka so i created a topic and i send it to logstash so when i run it i just get the message when i want see path,tags,input type....
here my filebeat config:

  hosts: [""]
  topic: "kafka-topic"
    string: '%{[message]}'
  - add_host_metadata: ~
  - add_cloud_metadata: ~

this one it gives me _id _index type @timestamp version message.

codec.json it gives me all data but i dont want it in format json


input {
kafka {
      bootstrap_servers => ""
      topics => "kafka-topic"
elasticsearch {
      hosts => [""]
      index => "kafka-%{+YYYY.MM.dd}"

Should i add somthing or ... ?

Any help would be sincerely appreciate!

Same problem :confused: i get just 6 field i cant find @metadata when i use
string: '%{[message]}'

If you are using codec.format as a string, you will need to specify the fields you want to be shipped, like the example in the documentation.

The add_host_metadata will add host.* fields, and the add_cloud_metadata will add cloud.* fields.

So you should try to use:

  string: '%{[host]} %{[cloud]} %{[message]}'
1 Like

Thank you very much for the reply!
I tried this but the %{[cloud]} didn't work without any error...
So I tried

  string: '%{[host]} %{[message]}'

but it gives me host like json format under message, there is no possibility to add each one on field like filebeat output to Elasticsearch?

I mean like this maybe?

The most important fields that i want are log.file.path and tags if it's possible please!


Which cloud provider are you using? You can check if your provider is supported in the documentation.

The host is a json object, if you do not want to add it as a json you will need to add every field individually.

Like this:

  string: '%{[host][os][platform]} %{[host][os][version]} %{[message]}'
1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.