How to disable instrumentation just for kafka producers (but not consumers) on APM java agent

spekdrum · November 22, 2021, 10:26am

Hi,
I now I can disable instrumentation for kafka events with disable_instrumentations property like this:

disable_instrumentations=kafka

But this removes all related to Kafka. I would like to disable instrumentation only when producing, but keep the consumers instrumented. The reason is because we have an app that produces tons of messages per every consumed message and we would like to focus instrumentation only on the consumer processes.

I think the responsible class of this is the following:

github.com

elastic/apm-agent-java/blob/master/apm-agent-plugins/apm-kafka-plugin/apm-kafka-headers-plugin/src/main/java/co/elastic/apm/agent/kafka/KafkaProducerHeadersInstrumentation.java

/*
 * Licensed to Elasticsearch B.V. under one or more contributor
 * license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright
 * ownership. Elasticsearch B.V. licenses this file to you under
 * the Apache License, Version 2.0 (the "License"); you may
 * not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *   http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing,
 * software distributed under the License is distributed on an
 * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 * KIND, either express or implied.  See the License for the
 * specific language governing permissions and limitations
 * under the License.
 */
package co.elastic.apm.agent.kafka;

This file has been truncated. show original

but I can't find any configuration related or a way to disable it.

Thanks!

Sylvain_Juge · November 22, 2021, 12:50pm

Hi @spekdrum ,

As you have discovered, this is not something that can be fixed using configuration.

The agent capture two types of objects for Kafka

transactions when consuming a message (what you are interested in)
spans when sending messages within a transaction (what you want to filter out)

Thus here one possible work-around would be to use an ingestion filter on ES/APM server to remove the messaging spans: doc.

However, removing the "send message" spans means that the Transaction trace view will be broken as it relies on those intermediate spans for parent/child relationships. However, that might also work quite well if you use-case does not depends heavily on the traces view, for example if most of the application activity is handling messages by writing something to database and the delegation that you want to ignore isn't the main thing you want to monitor for now.

system · December 13, 2021, 8:50am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.