# ElasticAgent 8.13 and Kafka **URL:** \ **Category:** Elastic Agent\ **Created:** [May 3, 2024, 11:36am UTC](https://discuss.elastic.co/t/elasticagent-8-13-and-kafka/358704 "2024-05-03T11:36:54Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![tapiojaa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tapiojaa/32/134828_2.png) [@tapiojaa](https://discuss.elastic.co/u/tapiojaa)\ **Post date:** [May 3, 2024, 11:36am UTC](https://discuss.elastic.co/t/elasticagent-8-13-and-kafka/358704/1 "2024-05-03T11:36:54Z")
I'm using ElasticAgent to output data into Kafka. When using version 8.13.3 there is an error: status: (FAILED) could not start output: failed to reload output: topic '%{[fields.kafka\_topic]}' is invalid, it must match '[a-zA-Z0-9.\_-]' accessing 'kafka' In Elastic-Agent.yml output is configured like this ```auto outputs: default: type: kafka hosts: - 'kafkaextsand1.xyz.com:9094' client_id: test-topic version: 1.0.0 sasl: mechanism: SCRAM-SHA-256 ssl: certificate_authorities: ['C:/certificate/ca.crt'] verification_mode: full username: "test-topic" password: "xxxxx" topic: '%{[fields.kafka_topic]}' headers: [] timeout: 30 broker_timeout: 30 codec.json: pretty: false partition: round_robin: group-events: 1 required_acks: 1 compression: gzip max_message_bytes: 1000000 ``` Input example. ```auto inputs: - id: x123 type: filestream paths: - C:/ProgramData/Microsoft/EdgeUpdate/Log/*.log fields: kafka_topic: test-client-logs ``` Same configuration is working perfectly when using ElasticAgent version 8.12.2 ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [May 3, 2024, 12:30pm UTC](https://discuss.elastic.co/t/elasticagent-8-13-and-kafka/358704/2 "2024-05-03T12:30:01Z")
You cannot use that anymore, this was removed when the Kafka output went GA on 8.13, before 8.13 it was in a beta state. It is mentioned in this [breakign change](https://www.elastic.co/guide/en/fleet/current/release-notes-8.13.0.html#breaking-changes-8.13.0). ---
**Author:** ![tapiojaa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tapiojaa/32/134828_2.png) [@tapiojaa](https://discuss.elastic.co/u/tapiojaa)\ **Post date:** [May 6, 2024, 10:26am UTC](https://discuss.elastic.co/t/elasticagent-8-13-and-kafka/358704/3 "2024-05-06T10:26:25Z")
Thanks. I should have read that by myself. For my point of view this change is bad, because I want to output different data to different Kafka topic (metrics, system logs, different file stream logs etc.) Is there some other way to do this? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [May 6, 2024, 12:30pm UTC](https://discuss.elastic.co/t/elasticagent-8-13-and-kafka/358704/4 "2024-05-06T12:30:24Z")
> [@tapiojaa](#): > > Is there some other way to do this? Probably only if you change your output to use the logstash output, then you could have some logic in logstash to send the data to different topics based on the content of the message. ---
**Author:** ![tapiojaa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tapiojaa/32/134828_2.png) [@tapiojaa](https://discuss.elastic.co/u/tapiojaa)\ **Post date:** [May 6, 2024, 12:58pm UTC](https://discuss.elastic.co/t/elasticagent-8-13-and-kafka/358704/5 "2024-05-06T12:58:18Z")
I think I figured out how to do this. I'll create multiple Kafka outputs, where everything else similar except name and topic. Then in input section refer to different output-name Example: ```auto filestream1: type: kafka hosts: - 'kafkainttest1.xyz.fi:9092' client_id: test-vm version: 1.0.0 sasl: mechanism: SCRAM-SHA-256 ssl: certificate_authorities: ["C:/certificate/ca.crt"] verification_mode: full username: "test-vm" password: "xxxzxzz" topic: "test-vm-polku4" headers: [] timeout: 30 broker_timeout: 30 codec.json: pretty: false partition: round_robin: group-events: 1 required_acks: 1 compression: gzip max_message_bytes: 1000000 ``` ```auto inputs: - id: x1234 type: filestream paths: - C:/programdata/Microsoft/Windows Defender/Support/*.log use_output: filestream1 ``` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [May 6, 2024, 1:10pm UTC](https://discuss.elastic.co/t/elasticagent-8-13-and-kafka/358704/6 "2024-05-06T13:10:21Z")
> [@tapiojaa](#): > > I'll create multiple Kafka outputs, where everything else similar except name and topic. It won't work, you cannot have more then one output, if you have only the last one will work. Elastic Agent and the Beats only support having **one** output. EDIT: It seems that it may work if you are using the Elastic Agent on standalone mode, but you will need to test it. Also, not sure if this requires a license or not, it looks similar to having different outputs to different policies, which requires a license. ---
**Author:** ![tapiojaa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tapiojaa/32/134828_2.png) [@tapiojaa](https://discuss.elastic.co/u/tapiojaa)\ **Post date:** [May 6, 2024, 1:22pm UTC](https://discuss.elastic.co/t/elasticagent-8-13-and-kafka/358704/7 "2024-05-06T13:22:36Z")
I've license. We're about to implement ECK chart, Elastic Agent and Fleet. At the moment we are using Bitnami Chart, Filebeat- and Winlogbeat agents ---
**Author:** ![tapiojaa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tapiojaa/32/134828_2.png) [@tapiojaa](https://discuss.elastic.co/u/tapiojaa)\ **Post date:** [May 7, 2024, 5:52am UTC](https://discuss.elastic.co/t/elasticagent-8-13-and-kafka/358704/8 "2024-05-07T05:52:27Z")
> [@leandrojmp](#): > > It won't work, you cannot have more then one output, if you have only the last one will work. > > Elastic Agent and the Beats only support having **one** output. What do mean by logstash output? I only have Kafka that is ingesting data ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [May 7, 2024, 1:04pm UTC](https://discuss.elastic.co/t/elasticagent-8-13-and-kafka/358704/9 "2024-05-07T13:04:11Z")
> [@tapiojaa](#): > > What do mean by logstash output? I mentioned Logstash because when using filebeat/elastic agent you cannot have multiple outputs for the same input, so the solution is to use a logstash output and have it direct the logs according to some logic. I do not use the Elastic Agent on stand alone mode, but it seems that in your case you will have multiple inputs and each input will have one different output, I'm not sure this works, you will need to test it. ---
**Author:** ![tapiojaa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tapiojaa/32/134828_2.png) [@tapiojaa](https://discuss.elastic.co/u/tapiojaa)\ **Post date:** [May 7, 2024, 1:22pm UTC](https://discuss.elastic.co/t/elasticagent-8-13-and-kafka/358704/10 "2024-05-07T13:22:24Z")
Test wasn't successful. I need to stick version 8.12 until I found some other solution.