# Labels are getting added to only transaction event not to span event

**URL:** <https://discuss.elastic.co/t/labels-are-getting-added-to-only-transaction-event-not-to-span-event/201069>\
**Category:** APM\
**Tags:** java, server\
**Created:** [September 25, 2019, 2:34pm UTC](https://discuss.elastic.co/t/labels-are-getting-added-to-only-transaction-event-not-to-span-event/201069 "2019-09-25T14:34:44Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![ramya13](https://avatars.discourse-cdn.com/v4/letter/r/a5b964/32.png) [@ramya13](https://discuss.elastic.co/u/ramya13)\
**Post date:** [September 25, 2019, 2:34pm UTC](https://discuss.elastic.co/t/labels-are-getting-added-to-only-transaction-event-not-to-span-event/201069/1 "2019-09-25T14:34:44Z")

</div>

**Kibana version** : Not using Kibana

**Elasticsearch version** : 6.8.3

**APM Server version** : 7.3.2

**APM Agent language and version** : 1.9.0

**Browser version** : Version 77.0.3865.90

**Original install method (e.g. download page, yum, deb, from source, etc.) and version**: Installed from elastic repository of yum. version-7.3.2

**Fresh install or upgraded from other version?** fresh install

**Is there anything special in your setup?**  
having a single index instead of having indices. disabled the default template  
Below is the apm-server configuration:  
apm-server.yml

```auto
apm-server:
  host: 0.0.0.0:8200
output.elasticsearch:
  hosts:
  - 172.31.40.192:9200
  index: st-default_write
setup.template.enabled: false

```

# template used:

```auto
{
  "mappings": {
    "doc": {
      "properties": {
        "time": {
              "type": "date",
              "format": "epoch_second||strict_date_optional_time"
         }
      }
    }
  }
}

```

# agent configuration:

We are using Java spring boot petclinic application and instrumenting apm elastic agent in that application.  
We have downloaded latest apm jar of version 1.9.0 and instrumented by -javaagent flag with the below configuration

```auto
"nohup java -javaagent:"/home/devops/petclinic/apm.jar" -Delastic.apm.service_name=petclinic -Delastic.apm.server_urls=http://172.31.16.27:8200 -Delastic.apm.metrics_interval=0s -Delastic.apm.span_frames_min_duration=-1ms -Delastic.apm.global_labels=key1=aaa,key2=sss,key3=ddd,key4=fff -jar /home/devops/petclinic/petclinic.jar --spring.profiles.active=mysql &"

```

**Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**:

I have added global\_labels in the agent configuration via Java system properties as mentioned above. Expected to have those labels in all events but I am getting those global labels only in the transaction event not in any other events.

I have even tried to add labels to Span via public API but it doesn't works. Below is the method I have used to instrument API all .java files:

```auto
import co.elastic.apm.api.ElasticApm;
import co.elastic.apm.api.Span;
Span span = ElasticApm.currentSpan();
span.addLabel("_tag_appName", "Java-application");
span.addLabel("_tag_Name", "consumer_producer");

```

# Expected behavior:

should add labels which are defined in global\_labels to all events.

# Actual behavior:

Added those global labels defined only in transaction event and not adding to those labels to other events.

**Steps to reproduce** :

1. Run the server with same configuration and exact version.
2. Run the apm agent with the given configuration using -javaagent flag
3. Generate some load on petclininc app.
4. Look into the event details in the elasticsearch.

# **Provide logs and/or server output (if relevant)**: logs of java agent:

```auto
2019-09-25 14:19:01.297 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving SPAN event (sequence 30151)
2019-09-25 14:19:01.297 [apm-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'WelcomeController#welcome' 00-36150e9c4a4e9bac9807ea362a237d01-44653bf577e37aef-01 (29059463) (1)
2019-09-25 14:19:01.297 [apm-reporter] TRACE co.elastic.apm.agent.impl.transaction.AbstractSpan - decrementing references at
java.lang.RuntimeException: This is an expected exception. Is just used to record where the reference count has been decremented.
        at co.elastic.apm.agent.impl.transaction.AbstractSpan.decrementReferences(AbstractSpan.java:390)
        at co.elastic.apm.agent.impl.transaction.Span.decrementReferences(Span.java:250)
        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.writeEvent(IntakeV2ReportingEventHandler.java:170)
        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.handleEvent(IntakeV2ReportingEventHandler.java:141)
        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:116)
        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:50)
        at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)
        at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.lang.Thread.run(Thread.java:748)
2019-09-25 14:19:01.297 [apm-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'DispatcherServlet#render welcome' 00-36150e9c4a4e9bac9807ea362a237d01-acb0608ebf816229-01 (5f1a4b88) (0)
2019-09-25 14:19:01.297 [apm-reporter] TRACE co.elastic.apm.agent.impl.transaction.AbstractSpan - decrementing references at
java.lang.RuntimeException: This is an expected exception. Is just used to record where the reference count has been decremented.
        at co.elastic.apm.agent.impl.transaction.AbstractSpan.decrementReferences(AbstractSpan.java:390)
        at co.elastic.apm.agent.impl.transaction.Span.decrementReferences(Span.java:252)
        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.writeEvent(IntakeV2ReportingEventHandler.java:170)
        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.handleEvent(IntakeV2ReportingEventHandler.java:141)
        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:116)
        at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:50)
        at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)
        at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
        at java.lang.Thread.run(Thread.java:748)
2019-09-25 14:19:01.297 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving TRANSACTION event (sequence 30152)
2019-09-25 14:19:01.298 [apm-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'WelcomeController#welcome' 00-36150e9c4a4e9bac9807ea362a237d01-44653bf577e37aef-01 (29059463) (0)

```

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [September 25, 2019, 3:00pm UTC](https://discuss.elastic.co/t/labels-are-getting-added-to-only-transaction-event-not-to-span-event/201069/2 "2019-09-25T15:00:00Z")

</div>

Hi and thanks for your detailed description.

It's true that the APM Server does not add the global labels to spans, only to transactions, errors and metrics. IIRC, that's basically to save space. This is the related APM Server issue: [https://github.com/elastic/apm/issues/48](https://github.com/elastic/apm/issues/48)

Could you describe why you need to have the labels on spans as well?

---

<div class="post-metadata">

**Author:** ![ramya13](https://avatars.discourse-cdn.com/v4/letter/r/a5b964/32.png) [@ramya13](https://discuss.elastic.co/u/ramya13)\
**Post date:** [September 26, 2019, 4:56am UTC](https://discuss.elastic.co/t/labels-are-getting-added-to-only-transaction-event-not-to-span-event/201069/3 "2019-09-26T04:56:58Z")

</div>

Hi and thanks for the reply.  
I am trying to develop a monitoring solution using elastic APM which shows the metrics in dashboard as like kibana. For that dashboard to come up, I should have labels in all events.  
And other reason is without unique lables, I cannot able to identify the spans which belongs to particular petclinic app in the scenario if I have more than one petclinic app running.

Please let me know if there is any other possibility to get lables in span event.

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [September 27, 2019, 1:21pm UTC](https://discuss.elastic.co/t/labels-are-getting-added-to-only-transaction-event-not-to-span-event/201069/4 "2019-09-27T13:21:50Z")

</div>

Could the [`environment`](https://www.elastic.co/guide/en/apm/agent/java/current/config-core.html#config-environment) option help you to disambiguate between your different services? Which global labels are you currently using?

---

<div class="post-metadata">

**Author:** ![ramya13](https://avatars.discourse-cdn.com/v4/letter/r/a5b964/32.png) [@ramya13](https://discuss.elastic.co/u/ramya13)\
**Post date:** [September 27, 2019, 1:43pm UTC](https://discuss.elastic.co/t/labels-are-getting-added-to-only-transaction-event-not-to-span-event/201069/5 "2019-09-27T13:43:05Z")

</div>

No. I wish to have multiple labels to a single service. For example:  
\_tag\_appName= Java-application  
\_tag\_Name=consumer\_producer  
and many more.  
Just environment option alone will not help me.

Please let me know if there is any other option to add multiple labels to a single service in all events.

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [September 27, 2019, 2:20pm UTC](https://discuss.elastic.co/t/labels-are-getting-added-to-only-transaction-event-not-to-span-event/201069/6 "2019-09-27T14:20:30Z")

</div>

Currently, there is no option to do that. I'll create an issue to re-think the decision that spans don't get populated with the global labels as this has come up already.

> [@ramya13](#):
>
> I am trying to develop a monitoring solution using elastic APM which shows the metrics in dashboard as like kibana.

I'm curious. What's missing in Kibana that you need to develop your own monitoring solution?

> [@ramya13](#):
>
> \_tag\_appName= Java-application  
> \_tag\_Name=consumer\_producer

Sounds like these fields are already available in the events, just in a different location:  
`_tag_appName= Java-application` is roughly equivalent to `agent.name: java`,  
`_tag_Name` would be equivalent to the [`service.name`](https://www.elastic.co/guide/en/apm/agent/java/current/config-core.html#config-service-name).

The agent also sends the hostname, Docker and Kubernetes metadata.

Given all those properties which help to disambiguate, what would still be left?

---

<div class="post-metadata">

**Author:** ![ramya13](https://avatars.discourse-cdn.com/v4/letter/r/a5b964/32.png) [@ramya13](https://discuss.elastic.co/u/ramya13)\
**Post date:** [September 30, 2019, 11:49am UTC](https://discuss.elastic.co/t/labels-are-getting-added-to-only-transaction-event-not-to-span-event/201069/7 "2019-09-30T11:49:37Z")

</div>

Hi,  
I have tried with environment option. But it didn't add environment value to span event. It adds only to the transaction event only.  
Is this the expected behavior of environment option?

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [September 30, 2019, 1:26pm UTC](https://discuss.elastic.co/t/labels-are-getting-added-to-only-transaction-event-not-to-span-event/201069/8 "2019-09-30T13:26:51Z")

</div>

Currently, the environment is indeed not added to spans. But that functionality will come in 7.4 which is going to be released soon. See also [https://github.com/elastic/apm-server/issues/2471](https://github.com/elastic/apm-server/issues/2471).

---

<div class="post-metadata">

**Author:** ![felixbarny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/felixbarny/32/27341_2.png) [@felixbarny](https://discuss.elastic.co/u/felixbarny)\
**Post date:** [September 30, 2019, 1:30pm UTC](https://discuss.elastic.co/t/labels-are-getting-added-to-only-transaction-event-not-to-span-event/201069/9 "2019-09-30T13:30:13Z")

</div>

You can follow the discussion whether spans should contain global labels here: [https://github.com/elastic/apm/issues/153](https://github.com/elastic/apm/issues/153)

---

<div class="post-metadata">

**Author:** ![ramya13](https://avatars.discourse-cdn.com/v4/letter/r/a5b964/32.png) [@ramya13](https://discuss.elastic.co/u/ramya13)\
**Post date:** [September 30, 2019, 2:58pm UTC](https://discuss.elastic.co/t/labels-are-getting-added-to-only-transaction-event-not-to-span-event/201069/10 "2019-09-30T14:58:01Z")

</div>

Ok. Thank you

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 21, 2019, 11:03am UTC](https://discuss.elastic.co/t/labels-are-getting-added-to-only-transaction-event-not-to-span-event/201069/11 "2019-10-21T11:03:16Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.