Full APM trace with Kakfa documentation

berryma4 · March 11, 2020, 12:47pm

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text

Kibana version:
v 7.5.2

Elasticsearch version:
7.5.2

APM Server version:
7.5.2

APM Agent language and version:
Java 1.14.0

Browser version:
Chrome

Original install method (e.g. download page, yum, deb, from source, etc.) and version:

Fresh install or upgraded from other version?

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

Steps to reproduce:
1.
2.
3.

Errors in browser console (if relevant):
N/A

Provide logs and/or server output (if relevant):
N/A

Is there documentation on how to setup APM tracing with kafka? I have setup the apm agent on two processes; one which grabs from a database using a hikari connection pool then using a kafka producer pushing to kafka, the second process uses a kafka consumer to push to elastic. All I see with this setup in the Kibana APM page are the kafka consumer events. I don't see the database, or pushing to elastic. I was hoping to see the timing of the full trace (db -> java process -> kafka -> java process -> elastic). Any documentation on this would be helpful.

my config:
export JAVA_OPTS={JAVA_OPTS}" -javaagent:/data/elastic-apm-agent.jar" export JAVA_OPTS={JAVA_OPTS}" -Delastic.apm.service_name=xxx-search"
export JAVA_OPTS={JAVA_OPTS}" -Delastic.apm.environment=test.xxx,test.xxx" export JAVA_OPTS={JAVA_OPTS}" -Delastic.apm.application_packages=com,org,xxx,java"
export JAVA_OPTS={JAVA_OPTS}" -Delastic.apm.server_urls=http://apm.elk.xxx.xxx:8200" export JAVA_OPTS={JAVA_OPTS}" -Delastic.apm.disable_instrumentations=mule"

Eyal_Koren · March 15, 2020, 5:37am

Our agent will only trace events that occur within traced transactions. A transaction would be the entry event on a JVM process that tells the agent to trace other events within its execution. The Java agent will start a transaction if a supported technology was used, for example- Servlets, some scheduling frameworks and consumers of messaging frameworks. It seems your producer side is not using any of those, in which case you can use our public API in order to start and stop a transaction manually.

The above explains why you DO see the consumer side of things, however, it does not explain why you don't see the writes to Elasticsearch. Note that if the consumer reads the events from the topic and not directly sends to Elasticsearch on the same thread (eg puts on a queue to be read by another thread) - then the agent cannot correlate that out of the box.
If reading from the topic and sending to Elasticsearch is done on the same thread, please add this info:

Which Elasticsearch client version are you using?
Which Kafka clients version are you using?
If you share your consumer code outlines, that may assist with analysis

berryma4 · March 16, 2020, 12:43pm

Elastic java client:
7.2.0
Kafka client:
2.1.1

But, you answered my question. Yes, I have a batch queue which kafka consumer writes, then the my queue consumer commits after a successful push to Elastic. I'm using the Elastic Retry object to send (which is also multithreaded):

github.com

elastic/elasticsearch/blob/master/server/src/main/java/org/elasticsearch/action/bulk/Retry.java

/*
 * Licensed to Elasticsearch under one or more contributor
 * license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright
 * ownership. Elasticsearch licenses this file to you under
 * the Apache License, Version 2.0 (the "License"); you may
 * not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing,
 * software distributed under the License is distributed on an
 * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 * KIND, either express or implied.  See the License for the
 * specific language governing permissions and limitations
 * under the License.
 */
package org.elasticsearch.action.bulk;

This file has been truncated. show original

So, from your comments, it looks like I have to modify code to make this work? Or is there an apm agent consumer config way to do this (ie. like the hikari connection pool)?

Separate question that might work for me, I have dynatrace trace ids. Is there a way to use those with configuration?

Thank you

Eyal_Koren · March 16, 2020, 1:29pm

Yes, it seems you will have to do some manual changes using our public API to make it all work.

Take a look at our example of propagating context through blocking queues.

Do you mean you are using the BulkProcessor API?

system · April 6, 2020, 9:29am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.