Java Agent: No data has been received from agents yet

Willliam · March 21, 2021, 4:50pm

Kibana version: 7.11.1

Elasticsearch version: 7.11.1

APM Server version: 7.11.2

APM Agent language and version: Java 1.21.0

Browser version: Google Chrome is up to date Version 89.0.4389.90 (Official Build) (64-bit)

Original install method (e.g. download page, yum, deb, from source, etc.) and version: download page elasticsearch-7.11.1-linux-x86_64.tar.gz

Fresh install or upgraded from other versions? Fresh install

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.

All Default

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

I installed ES, Kibana and APM Server on my local machine to set up APM. I've created a example spring boot App to test the Agent(java). Now when I go to Set Up Instructions on Kibana its able to check the status of APM server but for Agent status it says: No data has been received from agents yet

Steps to reproduce:

go to apm page in kibana
click on agent status
getting "No data has been received from agents yet"

Provide logs and/or server output (if relevant) :

I cannot find any error msg.

Felix_Roessel · March 21, 2021, 5:20pm

Welcome to our community @Willliam

Could you share how you configured the java agent? I guess using the java agent flag?

Willliam · March 22, 2021, 2:06am

vim elasticapm.properties file and edit the following lines

service_name=sample_apm
application_packages=accessing-data-mysql
server_url=http://tssaiops1:8200
environment=production
application_packages=org.example

run my spring boot app

java -javaagent:./elastic-apm-agent-1.21.0.jar -Delastic.apm.secret_token= -jar accessing-data-mysql-0.0.1-SNAPSHOT.jar

Felix_Roessel · March 22, 2021, 5:26am

1.) Do you have the properties file in the same directory as your agent.jar is?
2.) Are you sure the server is available on your localhost with the server url you specified?
3.) The logs from your app should show the start up process including the agent.. can you share them?

Willliam · March 22, 2021, 6:11am

1.) Do you have the properties file in the same directory as your agent.jar is?

Yes.

cd /root/data/apm/

ls -al

total 43788
drwxr-xr-x 2 root root 118 Mar 22 14:05 .
drwxr-xr-x 3 root root 17 Mar 14 18:32 ..
-rw-r--r-- 1 root root 36227488 Mar 21 16:18 accessing-data-mysql-0.0.1-SNAPSHOT.jar
-rw-r--r-- 1 root root 8603699 Mar 14 18:55 elastic-apm-agent-1.21.0.jar
-rw-r--r-- 1 root root 110 Mar 21 22:28 elasticapm.properties

cat elasticapm.properties

service_name=sample_apm
application_packages=com.example.accessingdatamysql
server_urls=http://tssaiops1:8200

java -javaagent:./elastic-apm-agent-1.21.0.jar -Delastic.apm.secret_token= -jar accessing-data-mysql-0.0.1-SNAPSHOT.jar

2021-03-22 14:07:14,492 [main] INFO co.elastic.apm.agent.util.JmxUtils - Found JVM-specific OperatingSystemMXBean interface: com.sun.management.OperatingSystemMXBean
2021-03-22 14:07:14,591 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.21.0 as sample_apm on Java 1.8.0_271 Runtime version: 1.8.0_271-b09 VM version: 25.271-b09 (Oracle Corporation) Linux 3.10.0-1127.el7.x86_64
2021-03-22 14:07:14,592 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - VM Arguments: [-javaagent:./elastic-apm-agent-1.21.0.jar, -Delastic.apm.secret_token=]
2021-03-22 14:07:17,686 [main] INFO co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
2021-03-22 14:07:17,895 [elastic-apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: { "build_date": "2021-03-06T04:41:35Z", "build_sha": "b706a93fac838d7ca44622d8d9686d2c3b3c8bde", "version": "7.11.2"}

. ____ _ __ _ _
/\ / ' __ _ () __ __ _ \ \ \
( ( )__ | '_ | '| | ' / ` | \ \ \
\/ )| |)| | | | | || (| | ) ) ) )
' || .__|| ||| |_, | / / / /
=========||==============|/=////
:: Spring Boot :: (v2.1.7.RELEASE)

2021-03-22 14:07:20.272 INFO 4604 --- [ 2021-03-22 14:07:20.275 INFO 4604 --- [ 2021-03-22 14:07:21.974 INFO 4604 --- [ 2021-03-22 14:07:22.062 INFO 4604 --- [ 2021-03-22 14:07:22.649 INFO 4604 --- [ 2021-03-22 14:07:23.228 INFO 4604 --- [ 2021-03-22 14:07:23.282 INFO 4604 --- [ 2021-03-22 14:07:23.282 INFO 4604 --- [ 2021-03-22 14:07:23.575 INFO 4604 --- [ 2021-03-22 14:07:23.575 INFO 4604 --- [ 2021-03-22 14:07:23.819 INFO 4604 --- [ 2021-03-22 14:07:24.864 INFO 4604 --- [ 2021-03-22 14:07:24.942 INFO 4604 --- [ name: default
...]
2021-03-22 14:07:25.243 INFO 4604 --- [ 2021-03-22 14:07:25.245 INFO 4604 --- [ 2021-03-22 14:07:25.447 INFO 4604 --- [ 2021-03-22 14:07:26.155 INFO 4604 --- [ 2021-03-22 14:07:26.864 INFO 4604 --- [ 2021-03-22 14:07:27.634 INFO 4604 --- [ 2021-03-22 14:07:27.809 WARN 4604 --- [ 2021-03-22 14:07:28,304 [main] INFO 2021-03-22 14:07:28.306 INFO 4604 --- [ 2021-03-22 14:07:28.323 INFO 4604 --- [ main] c.e.a.AccessingDataMysqlApplication : Starting AccessingDataMysqlApplication v0.0.1-SNAPSHOT on tssaiops1 with PID 4604 (/root/data/apm/accessing-data-mysql-0.0.1-SNAPSHOT.jar started by root in /root/data/apm)
main] c.e.a.AccessingDataMysqlApplication : No active profile set, falling back to default profiles: default
main] .s.d.r.c.RepositoryConfigurationDelegate : Bootstrapping Spring Data repositories in DEFAULT mode.
main] .s.d.r.c.RepositoryConfigurationDelegate : Finished Spring Data repository scanning in 73ms. Found 1 repository interfaces.
main] trationDelegate$BeanPostProcessorChecker : Bean 'org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration' of type [org.springframework.transaction.annotation.ProxyTransactionManagementConfiguration$$EnhancerBySpringCGLIB$$8550c927] is not eligible for getting processed by all BeanPostProcessors (for example: not eligible for auto-proxying)
main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat initialized with port(s): 8080 (http)
main] o.apache.catalina.core.StandardService : Starting service [Tomcat]
main] org.apache.catalina.core.StandardEngine : Starting Servlet engine: [Apache Tomcat/9.0.22]
main] o.a.c.c.C.[Tomcat].[localhost].[/] : Initializing Spring embedded WebApplicationContext
main] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 3236 ms
main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Starting...
main] com.zaxxer.hikari.HikariDataSource : HikariPool-1 - Start completed.
main] o.hibernate.jpa.internal.util.LogHelper : HHH000204: Processing PersistenceUnitInfo [
main] org.hibernate.Version : HHH000412: Hibernate Core {5.3.10.Final}
main] org.hibernate.cfg.Environment : HHH000206: hibernate.properties not found
main] o.hibernate.annotations.common.Version : HCANN000001: Hibernate Commons Annotations {5.0.4.Final}
main] org.hibernate.dialect.Dialect : HHH000400: Using dialect: org.hibernate.dialect.MySQL5Dialect
main] j.LocalContainerEntityManagerFactoryBean : Initialized JPA EntityManagerFactory for persistence unit 'default'
main] o.s.s.concurrent.ThreadPoolTaskExecutor : Initializing ExecutorService 'applicationTaskExecutor'
main] aWebConfiguration$JpaWebMvcConfiguration : spring.jpa.open-in-view is enabled by default. Therefore, database queries may be performed during view rendering. Explicitly configure spring.jpa.open-in-view to disable this warning
co.elastic.apm.agent.servlet.ServletVersionInstrumentation - Servlet container info = Apache Tomcat/9.0.22
main] o.s.b.w.embedded.tomcat.TomcatWebServer : Tomcat started on port(s): 8080 (http) with context path ''
main] c.e.a.AccessingDataMysqlApplication : Started AccessingDataMysqlApplication in 9.473 seconds (JVM running for 14.704)

2.) Are you sure the server is available on your localhost with the server url you specified?

Yes. ES server and apm server both are available. And I see that apm java agent can connect to apm server.

3.) The logs from your app should show the start up process including the agent.. can you share them?

Please check the above output.

Felix_Roessel · March 22, 2021, 7:30am

This looks good so far. You Java agent is able to connect to the APM Server.

Now we need to check APM server -> Elasticsearch..
Do you see any error in the APM server log?
Also run GET /_cat/indices/apm-* against your Elasticsearch, pls. (This shows whether we already have some APM indices in Elasticsearch)

Did your application executed any request already? AFAIK there is no data until the first requests runs after startup.

Willliam · March 22, 2021, 2:04pm

Do you see any error in the APM server log?
Also run GET /_cat/indices/apm-* against your Elasticsearch, pls. (This shows whether we already have some APM indices in Elasticsearch)

===> I can see some apm-* indices have been created.

Did your application executed any request already? AFAIK there is no data until the first requests runs after startup.

===> Yes, the application executed any request already.

By the way, I find a kibana issue(APM 7.11 services missing if no traffic). The issue seems like my problem. I will wait for ES 7.12.

system · April 12, 2021, 10:04am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.