Apm server publishing data to single kakfa topic partition

Elastic Observability APM
1

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text :slight_smile:

Kibana version:NA

Elasticsearch version: NA

APM Server version: 7.1.2

APM Agent language and version: Java

Browser version: NA

Original install method (e.g. download page, yum, deb, from source, etc.) and version: docker

Fresh install or upgraded from other version? NA

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? using kafka output

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant): I have enabled data publishing of transaction, span , beat on kafka. But problem is it is publishing all data to single kafka topic partition. It should be publishing to different partitions.

Steps to reproduce:

  1. Enabled kafka output.
    2.Did following configuration in apm-server.yaml

Errors in browser console (if relevant): NA
Provide logs and/or server output (if relevant):

output.kafka:
enabled: true
hosts: ["kafka:9092"]
topic: '%{[processor.event]}'
key: '%{[transaction.id]}'

How can i publish on different partitions of the kafka topic, so that its distributed ?

2

what does reachable_only does? somewhat not clear with difference in true/false

3

Hi!

You can find the answer in the official docs: https://www.elastic.co/guide/en/apm/server/current/kafka-output.html#topics-option-kafka

reachable_only will cause unavailable partitions to be skipped.

4

I have enabled the same but still transaction and spans are published only in 0th partition .

output.kafka:
enabled: true
hosts: ["kafka:9092"]
topic: '%{[processor.event]}'
key: '%{[transaction.id]}'
partition.round_robin:
reachable_only: true

5

Hi, you need a topics attribute as described in the docs, not simply topic.
Does that makes sense?

6

I'm only using transactions and span

7

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.