APM Java Agent - manual setup

Jurilz · December 14, 2020, 1:55pm

Kibana version : 7.10.0

Elasticsearch version : 7.10.0

APM Server version : 7.10.1

APM Agent language and version : Java - elastic-apm-agent-1.19.0.jar

Browser version : Firefox 83.0

Original install method (e.g. download page, yum, deb, from source, etc.) and version :

docker for elasticsearch, kibana, logstash and apm-server
apt-get for tomcat9

Good day,

I'm trying to monitore an apache tomcat9, which I set up locally under Ubuntu and can successfully reach under http://localhost:8080

So I naturally followed these instructions: https://www.elastic.co/guide/en/apm/agent/java/current/setup-javaagent.html

for the manuel setup with -javaagent flag.

I downloaded the elastic-apm-agent-1.19.0.jar from maven central.

Then I created bin/setenv.sh with the content:

export CATALINA_OPTS="$CATALINA_OPTS -javaagent:<path-to-my-elastic-apm-agent>.jar"
export CATALINA_OPTS="$CATALINA_OPTS -Delastic.apm.service_name=my-cool-service"
export CATALINA_OPTS="$CATALINA_OPTS -Delastic.apm.application_packages=org.example,org.another.example"
export CATALINA_OPTS="$CATALINA_OPTS -Delastic.apm.server_urls=http://localhost:8200"

Afterwards I made the setenv.sh executable with

chmod +x bin/setenv.sh

I executed the setenv.sh with sh setenv.sh and restarted the tomcat server with

sudo service tomcat9 restart

But Kibana still tells me that: No data has been received from agents yet. The apm-server is reachable under http://apm-server:8200.

apm-server. yml:

apm-server:
  host: "0.0.0.0:8200"

output.elasticsearch:
  hosts: ["https://es01:9200"]
  username: "elastic"
  password: <my_pw>
  ssl.certificate_authorities: ["/usr/share/elasticsearch/config/certificates/ca/ca.crt"]

The apm-server logs:

2020-12-14T13:11:01.980Z	INFO	[publisher_pipeline_output]	pipeline/output.go:143	Connecting to backoff(elasticsearch(https://es01:9200))
2020-12-14T13:11:01.980Z	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2020-12-14T13:11:01.981Z	INFO	[publisher]	pipeline/retry.go:223	  done
2020-12-14T13:11:02.014Z	INFO	[esclientleg]	eslegclient/connection.go:314	Attempting to connect to Elasticsearch version 7.10.0
2020-12-14T13:11:02.100Z	INFO	[license]	licenser/check.go:35	License is active for Basic
2020-12-14T13:11:02.100Z	INFO	[license]	licenser/check.go:22	Trial license active
2020-12-14T13:11:02.100Z	INFO	[license]	licenser/es_callback.go:51	Elasticsearch license: Trial
2020-12-14T13:11:02.102Z	INFO	[esclientleg]	eslegclient/connection.go:314	Attempting to connect to Elasticsearch version 7.10.0
2020-12-14T13:11:02.514Z	INFO	[index-management]	idxmgmt/manager.go:84	Overwrite ILM setup is disabled. 
2020-12-14T13:11:02.515Z	INFO	[index-management]	idxmgmt/manager.go:203	Set setup.template.name to 'apm-%{[observer.version]}'.
2020-12-14T13:11:02.515Z	INFO	[index-management]	idxmgmt/manager.go:205	Set setup.template.pattern to 'apm-%{[observer.version]}*'.
2020-12-14T13:11:02.545Z	INFO	template/load.go:97	Template apm-7.10.1 already exists and will not be overwritten.
2020-12-14T13:11:02.546Z	INFO	[index-management]	idxmgmt/manager.go:211	Finished loading index template.
2020-12-14T13:11:02.551Z	INFO	[index-management.ilm]	ilm/std.go:139	do not generate ilm policy: exists=true, overwrite=false
2020-12-14T13:11:02.551Z	INFO	[index-management]	idxmgmt/manager.go:242	ILM policy apm-rollover-30-days successfully loaded.
2020-12-14T13:11:02.576Z	INFO	template/load.go:97	Template apm-7.10.1-error already exists and will not be overwritten.
2020-12-14T13:11:02.577Z	INFO	[index-management]	idxmgmt/manager.go:223	Finished template setup for apm-7.10.1-error.
2020-12-14T13:11:02.584Z	INFO	[index-management]	idxmgmt/manager.go:259	Write alias apm-7.10.1-error successfully generated.
2020-12-14T13:11:02.605Z	INFO	template/load.go:97	Template apm-7.10.1-metric already exists and will not be overwritten.
2020-12-14T13:11:02.606Z	INFO	[index-management]	idxmgmt/manager.go:223	Finished template setup for apm-7.10.1-metric.
2020-12-14T13:11:02.612Z	INFO	[index-management]	idxmgmt/manager.go:259	Write alias apm-7.10.1-metric successfully generated.
2020-12-14T13:11:02.633Z	INFO	template/load.go:97	Template apm-7.10.1-profile already exists and will not be overwritten.
2020-12-14T13:11:02.634Z	INFO	[index-management]	idxmgmt/manager.go:223	Finished template setup for apm-7.10.1-profile.
2020-12-14T13:11:02.640Z	INFO	[index-management]	idxmgmt/manager.go:259	Write alias apm-7.10.1-profile successfully generated.
2020-12-14T13:11:02.661Z	INFO	template/load.go:97	Template apm-7.10.1-span already exists and will not be overwritten.
2020-12-14T13:11:02.661Z	INFO	[index-management]	idxmgmt/manager.go:223	Finished template setup for apm-7.10.1-span.
2020-12-14T13:11:02.667Z	INFO	[index-management]	idxmgmt/manager.go:259	Write alias apm-7.10.1-span successfully generated.
2020-12-14T13:11:02.694Z	INFO	template/load.go:97	Template apm-7.10.1-transaction already exists and will not be overwritten.
2020-12-14T13:11:02.694Z	INFO	[index-management]	idxmgmt/manager.go:223	Finished template setup for apm-7.10.1-transaction.
2020-12-14T13:11:02.700Z	INFO	[index-management]	idxmgmt/manager.go:259	Write alias apm-7.10.1-transaction successfully generated.
2020-12-14T13:11:02.700Z	INFO	[index-management]	idxmgmt/manager.go:126	Finished index management setup.
2020-12-14T13:11:02.706Z	INFO	[pipelines]	pipeline/register.go:53	Pipeline already registered: apm
2020-12-14T13:11:02.709Z	INFO	[pipelines]	pipeline/register.go:53	Pipeline already registered: apm_user_agent
2020-12-14T13:11:02.712Z	INFO	[pipelines]	pipeline/register.go:53	Pipeline already registered: apm_user_geo
2020-12-14T13:11:02.716Z	INFO	[pipelines]	pipeline/register.go:53	Pipeline already registered: apm_ingest_timestamp
2020-12-14T13:11:02.718Z	INFO	[pipelines]	pipeline/register.go:53	Pipeline already registered: apm_remove_span_metadata
2020-12-14T13:11:02.719Z	INFO	[pipelines]	pipeline/register.go:56	Registered Ingest Pipelines successfully.
2020-12-14T13:11:02.719Z	INFO	[publisher_pipeline_output]	pipeline/output.go:151	Connection to backoff(elasticsearch(https://es01:9200)) established

I not sure where my error lies, therefore I looking forward to any help.

felixbarny · December 14, 2020, 2:08pm

Hi and thanks for your questions.

Could you please upload the debug logs of the agent? See also https://www.elastic.co/guide/en/apm/agent/java/current/trouble-shooting.html for more details. By default, the logs should be in catalina.out but you can customize the location by setting log_file.

Jurilz · December 14, 2020, 2:23pm

Thanks for your support. The catalina.out logs are:

14-Dec-2020 14:03:07.869 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name:   Apache Tomcat/9.0.31 (Ubuntu)
14-Dec-2020 14:03:07.870 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:          Oct 20 2020 12:27:39 UTC
14-Dec-2020 14:03:07.870 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 9.0.31.0
14-Dec-2020 14:03:07.870 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:               Linux
14-Dec-2020 14:03:07.870 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            5.4.0-53-generic
14-Dec-2020 14:03:07.870 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:          amd64
14-Dec-2020 14:03:07.870 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             /usr/lib/jvm/java-8-oracle/jre
14-Dec-2020 14:03:07.870 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           1.8.0_45-b14
14-Dec-2020 14:03:07.870 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:            Oracle Corporation
14-Dec-2020 14:03:07.871 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         /var/lib/tomcat9
14-Dec-2020 14:03:07.871 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:         /usr/share/tomcat9
14-Dec-2020 14:03:07.905 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/var/lib/tomcat9/conf/logging.properties
14-Dec-2020 14:03:07.905 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
14-Dec-2020 14:03:07.906 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.awt.headless=true
14-Dec-2020 14:03:07.906 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
14-Dec-2020 14:03:07.906 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
14-Dec-2020 14:03:07.906 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
14-Dec-2020 14:03:07.906 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
14-Dec-2020 14:03:07.907 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/var/lib/tomcat9
14-Dec-2020 14:03:07.907 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/usr/share/tomcat9
14-Dec-2020 14:03:07.907 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/tmp
14-Dec-2020 14:03:07.907 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on >14-Dec-2020 14:03:08.738 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
14-Dec-2020 14:03:08.791 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [1,386] milliseconds
14-Dec-2020 14:03:08.834 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
14-Dec-2020 14:03:08.835 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.31 (Ubuntu)]
14-Dec-2020 14:03:08.848 INFO [main] org.apache.catalina.startup.HostConfig.deployDescriptor Deploying deployment descriptor [/etc/tomcat9/Catalina/localhost/manager.xml]
14-Dec-2020 14:03:08.873 WARNING [main] org.apache.catalina.startup.HostConfig.deployDescriptor The path attribute with value [/manager] in deployment descriptor [/etc/tomcat9/Catalina/localhost/manager.xml] has been i>14-Dec-2020 14:03:10.648 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were>14-Dec-2020 14:03:10.697 INFO [main] org.apache.catalina.startup.HostConfig.deployDescriptor Deployment of deployment descriptor [/etc/tomcat9/Catalina/localhost/manager.xml] has finished in [1,849] ms

I tried to change the log_level in the setenv.sh to:

export CATALINA_OPTS="$CATALINA_OPTS -Delastic.apm.log_level=DEBUG"

But I'm not sure if it's working.

Sylvain_Juge · December 14, 2020, 3:57pm

I'm not sure about it, but where did you created the setenv.sh file ? Is it in $CATALINA_BASE/bin/setenv.sh or $CATALINA_HOME/bin/setenv.sh

org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         /var/lib/tomcat9
org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:         /usr/share/tomcat9

Jurilz · December 14, 2020, 4:29pm

I feel a littlbe bit stupid, because I created the setenv.sh in /bin not in /usr/share/tomcat9/bin.

After I corrected this mistake and set up the kibana host, there seems to be a successfull start of the apm-agent and a connection to the apm-server.

catalina.log:

...
org.apache.catalina.startup.HostConfig.deployDescriptor The path attribute with value [/host-manager] in deployment descriptor [/etc/tomcat9/Catalina/localhost/host-manager.xml] has been ignored
14-Dec-2020 17:17:22.210 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
14-Dec-2020 17:17:22.213 INFO [main] org.apache.catalina.startup.HostConfig.deployDescriptor Deployment of deployment descriptor [/etc/tomcat9/Catalina/localhost/host-manager.xml] has finished in [1,626] ms
14-Dec-2020 17:17:22.215 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/var/lib/tomcat9/webapps/ROOT]
14-Dec-2020 17:17:24.538 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
14-Dec-2020 17:17:24.565 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory [/var/lib/tomcat9/webapps/ROOT] has finished in [2,350] ms
14-Dec-2020 17:17:24.579 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
14-Dec-2020 17:17:24.622 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in [7,655] milliseconds
14-Dec-2020 17:19:33.902 INFO [Thread-16] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-8080"]
14-Dec-2020 17:19:33.928 INFO [Thread-16] org.apache.catalina.core.StandardService.stopInternal Stopping service [Catalina]
14-Dec-2020 17:19:34.084 INFO [Thread-16] org.apache.coyote.AbstractProtocol.stop Stopping ProtocolHandler ["http-nio-8080"]
14-Dec-2020 17:19:34.097 INFO [Thread-16] org.apache.coyote.AbstractProtocol.destroy Destroying ProtocolHandler ["http-nio-8080"]
14-Dec-2020 17:19:48.316 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version name:   Apache Tomcat/9.0.31 (Ubuntu)
14-Dec-2020 17:19:48.317 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server built:          Oct 20 2020 12:27:39 UTC
14-Dec-2020 17:19:48.317 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Server version number: 9.0.31.0
14-Dec-2020 17:19:48.317 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Name:               Linux
14-Dec-2020 17:19:48.317 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log OS Version:            5.4.0-53-generic
14-Dec-2020 17:19:48.317 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Architecture:          amd64
14-Dec-2020 17:19:48.317 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Java Home:             /usr/lib/jvm/java-8-oracle/jre
14-Dec-2020 17:19:48.318 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Version:           1.8.0_45-b14
14-Dec-2020 17:19:48.318 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log JVM Vendor:            Oracle Corporation
14-Dec-2020 17:19:48.318 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_BASE:         /var/lib/tomcat9
14-Dec-2020 17:19:48.318 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log CATALINA_HOME:         /usr/share/tomcat9
14-Dec-2020 17:19:48.324 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.config.file=/var/lib/tomcat9/conf/logging.properties
14-Dec-2020 17:19:48.325 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
14-Dec-2020 17:19:48.325 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.awt.headless=true
14-Dec-2020 17:19:48.325 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djdk.tls.ephemeralDHKeySize=2048
14-Dec-2020 17:19:48.325 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.protocol.handler.pkgs=org.apache.catalina.webresources
14-Dec-2020 17:19:48.325 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dorg.apache.catalina.security.SecurityListener.UMASK=0027
14-Dec-2020 17:19:48.325 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -javaagent:/opt/docker/apm-agent/elastic-apm-agent-1.19.0.jar
14-Dec-2020 17:19:48.325 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Delastic.apm.service_name=my-cool-service
14-Dec-2020 17:19:48.325 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Delastic.apm.application_packages=org.example,org.another.example
14-Dec-2020 17:19:48.325 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Delastic.apm.server_urls=http://localhost:8200
14-Dec-2020 17:19:48.326 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Delastic.apm.log_level=DEBUG
14-Dec-2020 17:19:48.326 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dignore.endorsed.dirs=
14-Dec-2020 17:19:48.326 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.base=/var/lib/tomcat9
14-Dec-2020 17:19:48.326 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Dcatalina.home=/usr/share/tomcat9
14-Dec-2020 17:19:48.326 INFO [main] org.apache.catalina.startup.VersionLoggerListener.log Command line argument: -Djava.io.tmpdir=/tmp
14-Dec-2020 17:19:48.326 INFO [main] org.apache.catalina.core.AprLifecycleListener.lifecycleEvent The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: [/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib]
14-Dec-2020 17:19:48.776 INFO [main] org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler ["http-nio-8080"]
14-Dec-2020 17:19:48.821 INFO [main] org.apache.catalina.startup.Catalina.load Server initialization in [1,005] milliseconds
14-Dec-2020 17:19:48.878 INFO [main] org.apache.catalina.core.StandardService.startInternal Starting service [Catalina]
14-Dec-2020 17:19:48.878 INFO [main] org.apache.catalina.core.StandardEngine.startInternal Starting Servlet engine: [Apache Tomcat/9.0.31 (Ubuntu)]
14-Dec-2020 17:19:48.916 INFO [main] org.apache.catalina.startup.HostConfig.deployDescriptor Deploying deployment descriptor [/etc/tomcat9/Catalina/localhost/manager.xml]
14-Dec-2020 17:19:48.959 WARNING [main] org.apache.catalina.startup.HostConfig.deployDescriptor The path attribute with value [/manager] in deployment descriptor [/etc/tomcat9/Catalina/localhost/manager.xml] has been ignored
14-Dec-2020 17:19:51.531 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
14-Dec-2020 17:19:51.860 INFO [main] org.apache.catalina.startup.HostConfig.deployDescriptor Deployment of deployment descriptor [/etc/tomcat9/Catalina/localhost/manager.xml] has finished in [2,943] ms
14-Dec-2020 17:19:51.860 INFO [main] org.apache.catalina.startup.HostConfig.deployDescriptor Deploying deployment descriptor [/etc/tomcat9/Catalina/localhost/host-manager.xml]
14-Dec-2020 17:19:51.864 WARNING [main] org.apache.catalina.startup.HostConfig.deployDescriptor The path attribute with value [/host-manager] in deployment descriptor [/etc/tomcat9/Catalina/localhost/host-manager.xml] has been ignored
14-Dec-2020 17:19:53.452 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
14-Dec-2020 17:19:53.455 INFO [main] org.apache.catalina.startup.HostConfig.deployDescriptor Deployment of deployment descriptor [/etc/tomcat9/Catalina/localhost/host-manager.xml] has finished in [1,595] ms
14-Dec-2020 17:19:53.457 INFO [main] org.apache.catalina.startup.HostConfig.deployDirectory Deploying web application directory [/var/lib/tomcat9/webapps/ROOT]
14-Dec-2020 17:19:55.898 INFO [main] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
...

Jurilz · December 14, 2020, 4:30pm

The apm-server logs:

...
2020-12-14T16:15:23.533Z	INFO	[kibana]	kibana/connecting_client.go:81	Successfully obtained connection to Kibana.
2020-12-14T16:15:24.454Z	INFO	[publisher_pipeline_output]	pipeline/output.go:143	Connecting to backoff(elasticsearch(https://es01:9200))
2020-12-14T16:15:24.454Z	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
2020-12-14T16:15:24.454Z	INFO	[publisher]	pipeline/retry.go:223	  done
2020-12-14T16:15:24.484Z	INFO	[esclientleg]	eslegclient/connection.go:314	Attempting to connect to Elasticsearch version 7.10.0
2020-12-14T16:15:24.577Z	INFO	[license]	licenser/check.go:35	License is active for Basic
...
2020-12-14T16:15:24.578Z	INFO	[license]	licenser/es_callback.go:51	Elasticsearch license: Trial
2020-12-14T16:15:24.587Z	INFO	[pipelines]	pipeline/register.go:53	Pipeline already registered: apm
2020-12-14T16:15:24.591Z	INFO	[pipelines]	pipeline/register.go:53	Pipeline already registered: apm_user_agent
2020-12-14T16:15:24.594Z	INFO	[pipelines]	pipeline/register.go:53	Pipeline already registered: apm_user_geo
2020-12-14T16:15:24.603Z	INFO	[pipelines]	pipeline/register.go:53	Pipeline already registered: apm_ingest_timestamp
2020-12-14T16:15:24.611Z	INFO	[pipelines]	pipeline/register.go:53	Pipeline already registered: apm_remove_span_metadata
2020-12-14T16:15:24.612Z	INFO	[pipelines]	pipeline/register.go:56	Registered Ingest Pipelines successfully.
2020-12-14T16:15:24.616Z	INFO	[esclientleg]	eslegclient/connection.go:314	Attempting to connect to Elasticsearch version 7.10.0
2020-12-14T16:15:25.046Z	INFO	[index-management]	idxmgmt/manager.go:84	Overwrite ILM setup is disabled. 
2020-12-14T16:15:25.046Z	INFO	[index-management]	idxmgmt/manager.go:203	Set setup.template.name to 'apm-%{[observer.version]}'.
2020-12-14T16:15:25.046Z	INFO	[index-management]	idxmgmt/manager.go:205	Set setup.template.pattern to 'apm-%{[observer.version]}*'.
2020-12-14T16:15:25.078Z	INFO	template/load.go:97	Template apm-7.10.1 already exists and will not be overwritten.
2020-12-14T16:15:25.078Z	INFO	[index-management]	idxmgmt/manager.go:211	Finished loading index template.
2020-12-14T16:15:25.083Z	INFO	[index-management.ilm]	ilm/std.go:139	do not generate ilm policy: exists=true, overwrite=false
2020-12-14T16:15:25.083Z	INFO	[index-management]	idxmgmt/manager.go:242	ILM policy apm-rollover-30-days successfully loaded.
2020-12-14T16:15:25.127Z	INFO	template/load.go:97	Template apm-7.10.1-metric already exists and will not be overwritten.
2020-12-14T16:15:25.127Z	INFO	[index-management]	idxmgmt/manager.go:223	Finished template setup for apm-7.10.1-metric.
2020-12-14T16:15:25.136Z	INFO	[index-management]	idxmgmt/manager.go:259	Write alias apm-7.10.1-metric successfully generated.
2020-12-14T16:15:25.165Z	INFO	template/load.go:97	Template apm-7.10.1-profile already exists and will not be overwritten.
2020-12-14T16:15:25.165Z	INFO	[index-management]	idxmgmt/manager.go:223	Finished template setup for apm-7.10.1-profile.
2020-12-14T16:15:25.173Z	INFO	[index-management]	idxmgmt/manager.go:259	Write alias apm-7.10.1-profile successfully generated.
2020-12-14T16:15:25.205Z	INFO	template/load.go:97	Template apm-7.10.1-span already exists and will not be overwritten.
2020-12-14T16:15:25.205Z	INFO	[index-management]	idxmgmt/manager.go:223	Finished template setup for apm-7.10.1-span.
2020-12-14T16:15:25.217Z	INFO	[index-management]	idxmgmt/manager.go:259	Write alias apm-7.10.1-span successfully generated.
2020-12-14T16:15:25.248Z	INFO	template/load.go:97	Template apm-7.10.1-transaction already exists and will not be overwritten.
2020-12-14T16:15:25.248Z	INFO	[index-management]	idxmgmt/manager.go:223	Finished template setup for apm-7.10.1-transaction.
2020-12-14T16:15:25.255Z	INFO	[index-management]	idxmgmt/manager.go:259	Write alias apm-7.10.1-transaction successfully generated.
2020-12-14T16:15:25.278Z	INFO	template/load.go:97	Template apm-7.10.1-error already exists and will not be overwritten.
2020-12-14T16:15:25.278Z	INFO	[index-management]	idxmgmt/manager.go:223	Finished template setup for apm-7.10.1-error.
2020-12-14T16:15:25.288Z	INFO	[index-management]	idxmgmt/manager.go:259	Write alias apm-7.10.1-error successfully generated.
2020-12-14T16:15:25.288Z	INFO	[index-management]	idxmgmt/manager.go:126	Finished index management setup.
2020-12-14T16:15:25.288Z	INFO	[publisher_pipeline_output]	pipeline/output.go:151	Connection to backoff(elasticsearch(https://es01:9200)) established
2020-12-14T16:16:01.277Z	INFO	[request]	middleware/log_middleware.go:101	request accepted	{"request_id": "55d83265-22ad-4a15-95e0-fa654a81e161", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "172.22.0.1", "user-agent": "elasticapm-java/1.19.0", "event.duration": 10019554918, "response_code": 202}
...

Unfortunately Kibana tells me that the Agent Status is: No data has been received from agents yet

Jurilz · December 15, 2020, 11:41am

My current

amp-server.yml:

apm-server:
  host: "0.0.0.0:8200"

  kibana:
    host: "https://kib01:5601"
    username: "elastic"
    password: <my-pw>
    ssl.enabled: true
    ssl.certificate_authorities: ["/usr/share/elasticsearch/config/certificates/ca/ca.crt"]

output.elasticsearch:
  hosts: ["https://es01:9200"]
  username: "elastic"
  password: <my-pw>
  ssl.certificate_authorities: ["/usr/share/elasticsearch/config/certificates/ca/ca.crt"]

And the current /usr/share/tomcat9/bin/setenv.sh:

export CATALINA_OPTS="$CATALINA_OPTS -javaagent:/<path-to-my>/elastic-apm-agent-1.19.0.jar"
export CATALINA_OPTS="$CATALINA_OPTS -Delastic.apm.service_name=my-cool-service"
export CATALINA_OPTS="$CATALINA_OPTS -Delastic.apm.application_packages=org.example,org.another.example"
export CATALINA_OPTS="$CATALINA_OPTS -Delastic.apm.server_urls=http://localhost:8200"
export CATALINA_OPTS="$CATALINA_OPTS -Delastic.apm.log_level=DEBUG"

Although according to Kibana the agent status is "No data has been received from agents yet", I can load Kibana objects and launch an APM Dashboard in Kibana, with some information about requests per minute. I don't know how much it matters that my tomcat9 is currently "blank", i.e. that nothing happened to it after the install.

Sylvain_Juge · December 17, 2020, 9:08am

Hi @Jurilz,

Have you checked the apm-* indices in Kibana/Elasticsearch ?
When the app is running (but without any transactions), metrics are collected (about every minute by default if I remember correctly).

From the last line of server logs, we can see that the agent seem able to connect:

INFO	[request]	middleware/log_middleware.go:101	request accepted	{"request_id": "55d83265-22ad-4a15-95e0-fa654a81e161", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "172.22.0.1", "user-agent": "elasticapm-java/1.19.0", "event.duration": 10019554918, "response_code": 202}

Jurilz · December 17, 2020, 9:59am

The apm-* indices, especially the apm-7.10.1-metric-000001 look just fine.

I'm irritated by Kibana's saying that agent status is "No data has been received from agents yet".

Apart from that everything seems to work fine so far. Of course I will know more about the status, when I'm using the apm agent on a tomcat server, that is really used.

Sylvain_Juge · December 18, 2020, 3:42pm

I don't know about the heuristic that is used to detect if any data has been captured here, did this message still appears once you have executed a few transactions ?

If you started your application metrics will be collected even if there is no traffic on the app, unlike transactions and spans which are only captured passively when there is some requests made to the server (in your case tomcat).

Jurilz · December 21, 2020, 4:47pm

I just checked and this message appears even after a lot of transcation.

Apart from the message the APm agent and server seem to work fine.

system · January 11, 2021, 12:47pm

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.