Labels are getting added to only transaction event not to span event

Elastic Observability APM
,
1

Kibana version: Not using Kibana

Elasticsearch version: 6.8.3

APM Server version: 7.3.2

APM Agent language and version: 1.9.0

Browser version: Version 77.0.3865.90

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Installed from elastic repository of yum. version-7.3.2

Fresh install or upgraded from other version? fresh install

Is there anything special in your setup?
having a single index instead of having indices. disabled the default template
Below is the apm-server configuration:
apm-server.yml

apm-server:
  host: 0.0.0.0:8200
output.elasticsearch:
  hosts:
  - 172.31.40.192:9200
  index: st-default_write
setup.template.enabled: false

template used:

{
  "mappings": {
    "doc": {
      "properties": {
        "time": {
              "type": "date",
              "format": "epoch_second||strict_date_optional_time"
         }
      }
    }
  }
}

agent configuration:

We are using Java spring boot petclinic application and instrumenting apm elastic agent in that application.
We have downloaded latest apm jar of version 1.9.0 and instrumented by -javaagent flag with the below configuration

"nohup java -javaagent:"/home/devops/petclinic/apm.jar" -Delastic.apm.service_name=petclinic -Delastic.apm.server_urls=http://172.31.16.27:8200 -Delastic.apm.metrics_interval=0s -Delastic.apm.span_frames_min_duration=-1ms -Delastic.apm.global_labels=key1=aaa,key2=sss,key3=ddd,key4=fff -jar /home/devops/petclinic/petclinic.jar --spring.profiles.active=mysql &"

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

I have added global_labels in the agent configuration via Java system properties as mentioned above. Expected to have those labels in all events but I am getting those global labels only in the transaction event not in any other events.

I have even tried to add labels to Span via public API but it doesn't works. Below is the method I have used to instrument API all .java files:

import co.elastic.apm.api.ElasticApm;
import co.elastic.apm.api.Span;
Span span = ElasticApm.currentSpan();
span.addLabel("_tag_appName", "Java-application");
span.addLabel("_tag_Name", "consumer_producer");

Expected behavior:

should add labels which are defined in global_labels to all events.

Actual behavior:

Added those global labels defined only in transaction event and not adding to those labels to other events.

Steps to reproduce:

  1. Run the server with same configuration and exact version.
  2. Run the apm agent with the given configuration using -javaagent flag
  3. Generate some load on petclininc app.
  4. Look into the event details in the elasticsearch.

Provide logs and/or server output (if relevant):
logs of java agent:

2019-09-25 14:19:01.297 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving SPAN event (sequence 30151)
2019-09-25 14:19:01.297 [apm-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'WelcomeController#welcome' 00-36150e9c4a4e9bac9807ea362a237d01-44653bf577e37aef-01 (29059463) (1)
2019-09-25 14:19:01.297 [apm-reporter] TRACE co.elastic.apm.agent.impl.transaction.AbstractSpan - decrementing references at
java.lang.RuntimeException: This is an expected exception. Is just used to record where the reference count has been decremented.
        at co.elastic.apm.agent.impl.transaction.AbstractSpan.decrementReferences(AbstractSpan.java:390)
        at co.elastic.apm.agent.impl.transaction.Span.decrementReferences(Span.java:250)
        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.writeEvent(IntakeV2ReportingEventHandler.java:170)
        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.handleEvent(IntakeV2ReportingEventHandler.java:141)
        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:116)
        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:50)
        at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)
        at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.lang.Thread.run(Thread.java:748)
2019-09-25 14:19:01.297 [apm-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'DispatcherServlet#render welcome' 00-36150e9c4a4e9bac9807ea362a237d01-acb0608ebf816229-01 (5f1a4b88) (0)
2019-09-25 14:19:01.297 [apm-reporter] TRACE co.elastic.apm.agent.impl.transaction.AbstractSpan - decrementing references at
java.lang.RuntimeException: This is an expected exception. Is just used to record where the reference count has been decremented.
        at co.elastic.apm.agent.impl.transaction.AbstractSpan.decrementReferences(AbstractSpan.java:390)
        at co.elastic.apm.agent.impl.transaction.Span.decrementReferences(Span.java:252)
        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.writeEvent(IntakeV2ReportingEventHandler.java:170)
        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.handleEvent(IntakeV2ReportingEventHandler.java:141)
        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:116)
        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:50)
        at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)
        at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.lang.Thread.run(Thread.java:748)
2019-09-25 14:19:01.297 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving TRANSACTION event (sequence 30152)
2019-09-25 14:19:01.298 [apm-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'WelcomeController#welcome' 00-36150e9c4a4e9bac9807ea362a237d01-44653bf577e37aef-01 (29059463) (0)
2

Hi and thanks for your detailed description.

It's true that the APM Server does not add the global labels to spans, only to transactions, errors and metrics. IIRC, that's basically to save space. This is the related APM Server issue: https://github.com/elastic/apm/issues/48

Could you describe why you need to have the labels on spans as well?

3

Hi and thanks for the reply.
I am trying to develop a monitoring solution using elastic APM which shows the metrics in dashboard as like kibana. For that dashboard to come up, I should have labels in all events.
And other reason is without unique lables, I cannot able to identify the spans which belongs to particular petclinic app in the scenario if I have more than one petclinic app running.

Please let me know if there is any other possibility to get lables in span event.

4

Could the environment option help you to disambiguate between your different services? Which global labels are you currently using?

5

No. I wish to have multiple labels to a single service. For example:
_tag_appName= Java-application
_tag_Name=consumer_producer
and many more.
Just environment option alone will not help me.

Please let me know if there is any other option to add multiple labels to a single service in all events.

6

Currently, there is no option to do that. I'll create an issue to re-think the decision that spans don't get populated with the global labels as this has come up already.

I'm curious. What's missing in Kibana that you need to develop your own monitoring solution?

Sounds like these fields are already available in the events, just in a different location:
_tag_appName= Java-application is roughly equivalent to agent.name: java,
_tag_Name would be equivalent to the service.name.

The agent also sends the hostname, Docker and Kubernetes metadata.

Given all those properties which help to disambiguate, what would still be left?

7

Hi,
I have tried with environment option. But it didn't add environment value to span event. It adds only to the transaction event only.
Is this the expected behavior of environment option?

8

Currently, the environment is indeed not added to spans. But that functionality will come in 7.4 which is going to be released soon. See also https://github.com/elastic/apm-server/issues/2471.

9

You can follow the discussion whether spans should contain global labels here: https://github.com/elastic/apm/issues/153

10

Ok. Thank you

11

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.