Elastic Agent otel mode - kafka exporter

Elastic Observability
1

Hi all,

Ive got some concern about OpenTelemetry integration or so called "otel" mode for Elastic Agent.

According to github page: elastic-agent/internal/pkg/otel/README.md at main · elastic/elastic-agent · GitHub

there should be kafka exporter available. However when I configure such exporter I get following error:

root@mymachine:/opt/elastic-agent-8.17.3-linux-x86_64# ./elastic-agent otel --config ./otel.yml
Starting in otel mode
failed to get config: cannot unmarshal the configuration: decoding failed due to the following error(s):

error decoding 'exporters': unknown type: "kafka" for id: "kafka/otel" (valid values: [file elasticsearch loadbalancing otlphttp otlp debug])

I tried 8.17.1, 8.17.2, 8.17.3 versions already.

Did anyone face similar issue?
I know its in tech preview. Dont want to create github issue yet, maybe Im doing something wrong.

2

Hi @logmanp Welcome to the community.

Perhaps if you shared your otel.yml we might be able to help

3

@stephenb

Hi, Im pasting otel.yml - exporter part/section as here problem appears.
With elasticsearch exporters all works fine. When I add kafka definition it all breaks with error given.

exporters:
  # Exporter to send logs and metrics to Elasticsearch
  elasticsearch/ecs:
    endpoints: ["https://fb380dc5631e41a9b52213b48a31234.ece.domain.changed.net:9243"]
    api_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==
    mapping:
      mode: ecs
    logs_dynamic_index:
      enabled: true
    metrics_dynamic_index:
      enabled: true
    traces_dynamic_index:
      enabled: true
  elasticsearch/otel:
    endpoints: [ "https://fb380dc5631e41a9b52213b48a31234.ece.domain.changed.net:9243" ]
    api_key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==
    mapping:
      mode: otel
    logs_dynamic_index:
      enabled: true
    metrics_dynamic_index:
      enabled: true
    traces_dynamic_index:
      enabled: true
  kafka/otel:
    brokers:
      - broker1:9092
      - broker2:9092
      - broker3:9092
    topic: otel-poc
    protocol_version: 3.2.0
    auth:
      tls:
        insecure: false
        insecure_skip_verify: true
        ca_file: /opt/otel/elastic-otel/certs/ca.cer
        cert_file: /opt/otel/elastic-otel/certs/client.crt
        key_file: /opt/otel/elastic-otel/certs/client.key

I have kafka added under service section as well.

4

Hi @logmanp

I did some checking if you look at the actual 8.17.3 release kafka exporter is not supported yet

Which is also supported by the error message that you got above

You are looking at main Branch which not the correct documentation for the 8.17.3 release

I am not sure when the Kafka exporter will be part of the distribution...

I do not see the 8.18 branch yet, not sure if it will be in that or not.

Keep and eye out

5

Hi @stephenb , thanks for quick reply. Yes that was what I suspected looking at error, didnt get through version specific docs though.

Im starting to watch next releases then as otel looks quite promising.

Thanks!

1 Like