Kibana version: Not using Kibana
Elasticsearch version: 6.8.3
APM Server version: 7.3.2
APM Agent language and version: 1.9.0
Browser version: Version 77.0.3865.90
Original install method (e.g. download page, yum, deb, from source, etc.) and version: Installed from elastic repository of yum. version-7.3.2
Fresh install or upgraded from other version? fresh install
Is there anything special in your setup?
having a single index instead of having indices. disabled the default template
Below is the apm-server configuration:
apm-server.yml
apm-server:
host: 0.0.0.0:8200
output.elasticsearch:
hosts:
- 172.31.40.192:9200
index: st-default_write
setup.template.enabled: false
template used:
{
"mappings": {
"doc": {
"properties": {
"time": {
"type": "date",
"format": "epoch_second||strict_date_optional_time"
}
}
}
}
}
agent configuration:
We are using Java spring boot petclinic application and instrumenting apm elastic agent in that application.
We have downloaded latest apm jar of version 1.9.0 and instrumented by -javaagent flag with the below configuration
"nohup java -javaagent:"/home/devops/petclinic/apm.jar" -Delastic.apm.service_name=petclinic -Delastic.apm.server_urls=http://172.31.16.27:8200 -Delastic.apm.metrics_interval=0s -Delastic.apm.span_frames_min_duration=-1ms -Delastic.apm.global_labels=key1=aaa,key2=sss,key3=ddd,key4=fff -jar /home/devops/petclinic/petclinic.jar --spring.profiles.active=mysql &"
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
I have added global_labels in the agent configuration via Java system properties as mentioned above. Expected to have those labels in all events but I am getting those global labels only in the transaction event not in any other events.
I have even tried to add labels to Span via public API but it doesn't works. Below is the method I have used to instrument API all .java files:
import co.elastic.apm.api.ElasticApm;
import co.elastic.apm.api.Span;
Span span = ElasticApm.currentSpan();
span.addLabel("_tag_appName", "Java-application");
span.addLabel("_tag_Name", "consumer_producer");
Expected behavior:
should add labels which are defined in global_labels to all events.
Actual behavior:
Added those global labels defined only in transaction event and not adding to those labels to other events.
Steps to reproduce:
- Run the server with same configuration and exact version.
- Run the apm agent with the given configuration using -javaagent flag
- Generate some load on petclininc app.
- Look into the event details in the elasticsearch.
Provide logs and/or server output (if relevant):
logs of java agent:
2019-09-25 14:19:01.297 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving SPAN event (sequence 30151)
2019-09-25 14:19:01.297 [apm-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'WelcomeController#welcome' 00-36150e9c4a4e9bac9807ea362a237d01-44653bf577e37aef-01 (29059463) (1)
2019-09-25 14:19:01.297 [apm-reporter] TRACE co.elastic.apm.agent.impl.transaction.AbstractSpan - decrementing references at
java.lang.RuntimeException: This is an expected exception. Is just used to record where the reference count has been decremented.
at co.elastic.apm.agent.impl.transaction.AbstractSpan.decrementReferences(AbstractSpan.java:390)
at co.elastic.apm.agent.impl.transaction.Span.decrementReferences(Span.java:250)
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.writeEvent(IntakeV2ReportingEventHandler.java:170)
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.handleEvent(IntakeV2ReportingEventHandler.java:141)
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:116)
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:50)
at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)
at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
at java.lang.Thread.run(Thread.java:748)
2019-09-25 14:19:01.297 [apm-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'DispatcherServlet#render welcome' 00-36150e9c4a4e9bac9807ea362a237d01-acb0608ebf816229-01 (5f1a4b88) (0)
2019-09-25 14:19:01.297 [apm-reporter] TRACE co.elastic.apm.agent.impl.transaction.AbstractSpan - decrementing references at
java.lang.RuntimeException: This is an expected exception. Is just used to record where the reference count has been decremented.
at co.elastic.apm.agent.impl.transaction.AbstractSpan.decrementReferences(AbstractSpan.java:390)
at co.elastic.apm.agent.impl.transaction.Span.decrementReferences(Span.java:252)
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.writeEvent(IntakeV2ReportingEventHandler.java:170)
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.handleEvent(IntakeV2ReportingEventHandler.java:141)
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:116)
at co.elastic.apm.agent.report.IntakeV2ReportingEventHandler.onEvent(IntakeV2ReportingEventHandler.java:50)
at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:168)
at co.elastic.apm.agent.shaded.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:125)
at java.lang.Thread.run(Thread.java:748)
2019-09-25 14:19:01.297 [apm-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Receiving TRANSACTION event (sequence 30152)
2019-09-25 14:19:01.298 [apm-reporter] DEBUG co.elastic.apm.agent.impl.transaction.AbstractSpan - decrement references to 'WelcomeController#welcome' 00-36150e9c4a4e9bac9807ea362a237d01-44653bf577e37aef-01 (29059463) (0)