# Filebeat output kafka (processors)

**URL:** <https://discuss.elastic.co/t/filebeat-output-kafka-processors/299071>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 8, 2022, 10:12am UTC](https://discuss.elastic.co/t/filebeat-output-kafka-processors/299071 "2022-03-08T10:12:44Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![alex\_vermex](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_vermex/32/101267_2.png) [@alex\_vermex](https://discuss.elastic.co/u/alex_vermex)\
**Post date:** [March 8, 2022, 10:12am UTC](https://discuss.elastic.co/t/filebeat-output-kafka-processors/299071/1 "2022-03-08T10:12:44Z")

</div>

Hi,  
I have a problem i think in "processors" , I use the output kafka so i created a topic and i send it to logstash so when i run it i just get the message when i want see path,tags,input type....  
here my filebeat config:

```auto
output.kafka:
  hosts: ["127.0.0.1:9092"]
  topic: "kafka-topic"
  codec.format:
    string: '%{[message]}'
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

```

this one it gives me \_id \_index type @timestamp version message.

codec.json it gives me all data but i dont want it in format json

logstash.conf

```auto
input {
kafka {
      bootstrap_servers => "127.0.0.1:9092"
      topics => "kafka-topic"
    }
}
output{
elasticsearch {
      hosts => ["127.0.0.1:9200"]
      index => "kafka-%{+YYYY.MM.dd}"
  }
}

```

Should i add somthing or ... ?

Any help would be sincerely appreciate!  
Thanks!

---

<div class="post-metadata">

**Author:** ![Abderraouf\_ZAYEN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abderraouf_zayen/32/98042_2.png) [@Abderraouf\_ZAYEN](https://discuss.elastic.co/u/Abderraouf_ZAYEN)\
**Post date:** [March 8, 2022, 2:29pm UTC](https://discuss.elastic.co/t/filebeat-output-kafka-processors/299071/2 "2022-03-08T14:29:38Z")

</div>

Same problem 😕 i get just 6 field i cant find @metadata when i use  
codec.format:  
string: '%{[message]}'

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [March 8, 2022, 2:42pm UTC](https://discuss.elastic.co/t/filebeat-output-kafka-processors/299071/3 "2022-03-08T14:42:58Z")

</div>

If you are using `codec.format` as a `string`, you will need to specify the fields you want to be shipped, like the [example in the documentation](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-output-codec.html#configuration-output-codec).

The `add_host_metadata` will add `host.*` fields, and the `add_cloud_metadata` will add `cloud.*` fields.

So you should try to use:

```auto
codec.format:
  string: '%{[host]} %{[cloud]} %{[message]}'

```

---

<div class="post-metadata">

**Author:** ![alex\_vermex](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_vermex/32/101267_2.png) [@alex\_vermex](https://discuss.elastic.co/u/alex_vermex)\
**Post date:** [March 8, 2022, 3:55pm UTC](https://discuss.elastic.co/t/filebeat-output-kafka-processors/299071/4 "2022-03-08T15:55:42Z")

</div>

Thank you very much for the reply!  
I tried this but the %{[cloud]} didn't work without any error...  
So I tried

```auto
codec.format:
  string: '%{[host]} %{[message]}'

```

but it gives me host like json format under message, there is no possibility to add each one on field like filebeat output to Elasticsearch?

I mean like this maybe?

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/7/a7f9e277087254be100a1ca74410136dc2fb7095.png)

The most important fields that i want are log.file.path and tags if it's possible please!

Thanks!

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [March 8, 2022, 5:13pm UTC](https://discuss.elastic.co/t/filebeat-output-kafka-processors/299071/5 "2022-03-08T17:13:30Z")

</div>

Which cloud provider are you using? You can check if your provider is [supported](https://www.elastic.co/guide/en/beats/filebeat/current/add-cloud-metadata.html#add-cloud-metadata) in the documentation.

The `host` is a json object, if you do not want to add it as a json you will need to add every field individually.

Like this:

```auto
codec.format:
  string: '%{[host][os][platform]} %{[host][os][version]} %{[message]}'

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 5, 2022, 7:13pm UTC](https://discuss.elastic.co/t/filebeat-output-kafka-processors/299071/6 "2022-04-05T19:13:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.