ElasticAgent 8.13 and Kafka

Elastic Stack Elastic Agent
1

I'm using ElasticAgent to output data into Kafka. When using version 8.13.3 there is an error:

status: (FAILED) could not start output: failed to reload output: topic '%{[fields.kafka_topic]}' is invalid, it must match '[a-zA-Z0-9._-]' accessing 'kafka'

In Elastic-Agent.yml output is configured like this

outputs:
  default:
    type: kafka
    hosts:
      - 'kafkaextsand1.xyz.com:9094'
    client_id: test-topic
    version: 1.0.0
    sasl:
        mechanism: SCRAM-SHA-256
    ssl:
        certificate_authorities: ['C:/certificate/ca.crt']
        verification_mode: full
    username: "test-topic"
    password: "xxxxx"
    topic: '%{[fields.kafka_topic]}'
    headers: []
    timeout: 30
    broker_timeout: 30
    codec.json:
        pretty: false
    partition:
        round_robin:
            group-events: 1
    required_acks: 1
    compression: gzip
    max_message_bytes: 1000000

Input example.

inputs:
  - id: x123
    type: filestream
    paths:
        - C:/ProgramData/Microsoft/EdgeUpdate/Log/*.log
    fields:
        kafka_topic: test-client-logs

Same configuration is working perfectly when using ElasticAgent version 8.12.2

2

You cannot use that anymore, this was removed when the Kafka output went GA on 8.13, before 8.13 it was in a beta state.

It is mentioned in this breakign change.

1 Like
3

Thanks. I should have read that by myself. For my point of view this change is bad, because I want to output different data to different Kafka topic (metrics, system logs, different file stream logs etc.) Is there some other way to do this?

4

Probably only if you change your output to use the logstash output, then you could have some logic in logstash to send the data to different topics based on the content of the message.

5

I think I figured out how to do this. I'll create multiple Kafka outputs, where everything else similar except name and topic. Then in input section refer to different output-name

Example:

  filestream1:
    type: kafka
    hosts:
      - 'kafkainttest1.xyz.fi:9092'
    client_id: test-vm
    version: 1.0.0
    sasl:
        mechanism: SCRAM-SHA-256
    ssl:
        certificate_authorities: ["C:/certificate/ca.crt"]
        verification_mode: full
    username: "test-vm"
    password: "xxxzxzz"
    topic: "test-vm-polku4"
    headers: []
    timeout: 30
    broker_timeout: 30
    codec.json:
        pretty: false
    partition:
        round_robin:
            group-events: 1
    required_acks: 1
    compression: gzip
    max_message_bytes: 1000000  

inputs:
  - id: x1234
    type: filestream
    paths:
        - C:/programdata/Microsoft/Windows Defender/Support/*.log
    use_output: filestream1
6

It won't work, you cannot have more then one output, if you have only the last one will work.

Elastic Agent and the Beats only support having one output.

EDIT:

It seems that it may work if you are using the Elastic Agent on standalone mode, but you will need to test it.

Also, not sure if this requires a license or not, it looks similar to having different outputs to different policies, which requires a license.

7

I've license. We're about to implement ECK chart, Elastic Agent and Fleet. At the moment we are using Bitnami Chart, Filebeat- and Winlogbeat agents

8

What do mean by logstash output? I only have Kafka that is ingesting data

9

I mentioned Logstash because when using filebeat/elastic agent you cannot have multiple outputs for the same input, so the solution is to use a logstash output and have it direct the logs according to some logic.

I do not use the Elastic Agent on stand alone mode, but it seems that in your case you will have multiple inputs and each input will have one different output, I'm not sure this works, you will need to test it.

10

Test wasn't successful. I need to stick version 8.12 until I found some other solution.