Beginer error, sent data to APM but this is not showed on kibana

targino.cbranco · December 2, 2021, 8:35pm

Hi poeple!

I'm starting with ELK, and I get some problems, I installed the last version of Elasticsearch, kibana and apm server, on kibana apm pages, I can see successful test of connectivity with apm server, but agent don't send data.

KIbana 7.15.2:

Elasticsearch 7.15.2:

APM Server 7.15.2:

APM Agent java elastic-apm-agent-1.27.1.jar:

Chrome Versão 96.0.4664.45:

**Install by yum **:

**I started my java agent together with standalone application, by command line, in the agent log no error could be found, the service is created but not other data of my application is recorded.

I have searched for errors on logs and each component is connecting normally with others.

I don't know to where continue and needed to ask for help, bellow I'll put environment informations and I'll be so great if someone can help me. **:

Regards,

Targino

Sylvain_Juge · December 3, 2021, 8:21am

Hi @targino.cbranco , welcome to the forum !

You should probably start with the troubleshooting section of the documentation,
especially in your case the There is no data in the Kibana APM app seems relevant after investigating things at agent level.

targino.cbranco · December 3, 2021, 8:01pm

Yes, I started with rroubleshooting but could not find my problem, so I needed to come here try to get some help.

Sylvain_Juge · December 6, 2021, 8:20am

Hi,

Could you elaborate a bit on what you have done so far? What steps of the troubleshooting guide did you followed and which ones did not worked as expected ?
Providing agent/server logs would be relevant here.

targino.cbranco · December 17, 2021, 8:28pm

Hello @Sylvain_Juge sorry for delay! I was busy with others tasks, today I restarted it, and I rebuild my stack with the last version (7.16.1) using docker images.

When I start my java standalone apllication processador.jar the services is created, I recive data for JVM usage but don't recive data for traces.

I started my application with agent elastic-apm-agent-1.28.1.jar and put to log, I could see that when the application starts the agent get a exception.

2021-12-17 16:42:36,654 [main] DEBUG co.elastic.apm.agent.util.ExecutorUtils - A new thread named `elastic-apm-configuration-reloader` was created. The original context class loader of this thread (jdk.internal.loader.ClassLoaders$AppClassLoader@2c13da15) has been overridden
2021-12-17 16:42:36,667 [main] TRACE co.elastic.apm.agent.util.ExecutorUtils - Stack trace related to thread creation:
java.lang.Throwable: null
        at co.elastic.apm.agent.util.ExecutorUtils.logThreadCreation(ExecutorUtils.java:99) [elastic-apm-agent-1.28.1.jar:?]
        at co.elastic.apm.agent.util.ExecutorUtils$SingleNamedThreadFactory.newThread(ExecutorUtils.java:88) [elastic-apm-agent-1.28.1.jar:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.<init>(ThreadPoolExecutor.java:623) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:912) [?:?]
        at java.util.concurrent.ThreadPoolExecutor.ensurePrestart(ThreadPoolExecutor.java:1583) [?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:346) [?:?]
        at java.util.concurrent.ScheduledThreadPoolExecutor.scheduleAtFixedRate(ScheduledThreadPoolExecutor.java:632) [?:?]
        at co.elastic.apm.agent.impl.ElasticApmTracerBuilder.scheduleReloadAtRate(ElasticApmTracerBuilder.java:180) [elastic-apm-agent-1.28.1.jar:?]
        at co.elastic.apm.agent.impl.ElasticApmTracerBuilder.build(ElasticApmTracerBuilder.java:143) [elastic-apm-agent-1.28.1.jar:?]
        at co.elastic.apm.agent.impl.ElasticApmTracerBuilder.build(ElasticApmTracerBuilder.java:123) [elastic-apm-agent-1.28.1.jar:?]
        at co.elastic.apm.agent.bci.ElasticApmAgent.initialize(ElasticApmAgent.java:145) [elastic-apm-agent-1.28.1.jar:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at co.elastic.apm.agent.premain.AgentMain.loadAndInitializeAgent(AgentMain.java:149) [elastic-apm-agent-1.28.1.jar:1.28.1]
        at co.elastic.apm.agent.premain.AgentMain.init(AgentMain.java:93) [elastic-apm-agent-1.28.1.jar:1.28.1]
        at co.elastic.apm.agent.premain.AgentMain.premain(AgentMain.java:49) [elastic-apm-agent-1.28.1.jar:1.28.1]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
        at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513) [?:?]
        at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525) [?:?]
2021-12-17 16:42:36,817 [main] DEBUG co.elastic.apm.agent.util.ExecutorUtils - A new thread named `elastic-apm-metadata-0` was created. The original context class loader of this thread (jdk.internal.loader.ClassLoaders$AppClassLoader@2c13da15) has been overridden

This exception performs a lot of times and after the agent estabilish send data

{"metricset":{"timestamp":1639772522633000,"samples":{"system.process.cpu.total.norm.pct":{"value":0.002504382669671926},"jvm.memory.heap.used":{"value":355583792.0},"jvm.memory.non_heap.used":{"value":256823848.0},"jvm.memory.heap.max":{"value":7834959872.0},"jvm.gc.alloc":{"value":3162691952.0},"jvm.memory.non_heap.committed":{"value":262021120.0},"system.cpu.total.norm.pct":{"value":0.2706402871692128},"system.process.cgroup.memory.mem.usage.bytes":{"value":8411471872.0},"system.process.memory.size":{"value":12187881472.0},"jvm.memory.heap.committed":{"value":1264582656.0},"jvm.memory.non_heap.max":{"value":-1.0},"system.memory.actual.free":{"value":22912049152.0},"system.memory.total":{"value":31331909632.0},"jvm.thread.count":{"value":74.0}}}}
{"metricset":{"timestamp":1639772522635000,"tags":{"name":"G1 Old Gen"},"samples":{"jvm.memory.heap.pool.committed":{"value":1073741824.0},"jvm.memory.heap.pool.max":{"value":7834959872.0},"jvm.memory.heap.pool.used":{"value":192005936.0}}}}
{"metricset":{"timestamp":1639772522635000,"tags":{"name":"G1 Young Generation"},"samples":{"jvm.gc.time":{"value":2082.0},"jvm.gc.count":{"value":29.0}}}}
{"metricset":{"timestamp":1639772522635000,"tags":{"name":"G1 Old Generation"},"samples":{"jvm.gc.time":{"value":0.0},"jvm.gc.count":{"value":0.0}}}}
{"metricset":{"timestamp":1639772522635000,"tags":{"name":"G1 Eden Space"},"samples":{"jvm.memory.heap.pool.committed":{"value":179306496.0},"jvm.memory.heap.pool.max":{"value":-1.0},"jvm.memory.heap.pool.used":{"value":152043520.0}}}}
{"metricset":{"timestamp":1639772522635000,"tags":{"name":"G1 Survivor Space"},"samples":{"jvm.memory.heap.pool.committed":{"value":11534336.0},"jvm.memory.heap.pool.max":{"value":-1.0},"jvm.memory.heap.pool.used":{"value":11534336.0}}}}

2021-12-17 17:22:12,653 [elastic-apm-server-reporter] DEBUG co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Flushing 2192 uncompressed 758 compressed bytes
2021-12-17 17:22:12,734 [elastic-apm-circuit-breaker] TRACE co.elastic.apm.agent.impl.circuitbreaker.CircuitBreaker - Scheduling next stress monitor polling in 5000s
2021-12-17 17:22:17,734 [elastic-apm-circuit-breaker] TRACE co.elastic.apm.agent.impl.circuitbreaker.CircuitBreaker - Scheduling next stress monitor polling in 5000s
2021-12-17 17:22:22,735 [elastic-apm-circuit-breaker] TRACE co.elastic.apm.agent.impl.circuitbreaker.CircuitBreaker - Scheduling next stress monitor polling in 5000s
2021-12-17 17:22:27,735 [elastic-apm-circuit-breaker] TRACE co.elastic.apm.agent.impl.circuitbreaker.CircuitBreaker - Scheduling next stress monitor polling in 5000s
2021-12-17 17:22:28,019 [elastic-apm-configuration-reloader] DEBUG co.elastic.apm.agent.impl.ElasticApmTracerBuilder - Beginning scheduled configuration reload (interval is 30 sec)...

Really I don't know how to solve it, and I can't to find a solution on the forum, I tryed to execute my application with java 8 and 11, and I get the same problem.

If you could help me I will be greatfull

targino.cbranco · December 17, 2021, 9:00pm

Hello again @Sylvain_Juge I did a test with a Python Flask application exemple and everything working very well the problem is only with java agent

Thanks for any help.

Sylvain_Juge · December 20, 2021, 8:13am

Hi @targino.cbranco ,

Actually the initial "ERROR" that you reported about 2 days ago is expected, and it's not an error.
This stack trace is just to help generate debug information related to thread creation by the agent, it's only included in the logs because you have used log_level=TRACE, where log_level=DEBUG should be enough for diagnosis for now.

One thing worth noting is that agent only passively collects activity on the application, when testing you have to make sure to execute a few transactions on the application to get traces. When doing so. When doing so, you should see DEBUG messages in the log with the following terms: startTransaction, startSpan, endTransaction and endSpan.

Also, you have to make sure that your application is using a framework/library that we currently support. See supported technologies for details.

targino.cbranco · December 29, 2021, 1:59pm

Hi @Sylvain_Juge

Theres no problem with my enviroment, now I have it working very, I found the real problem, I was testing with non http applications, so when put APM to monitore some API this started to get data normaly.

Now I would to know if could get data about performance in non http applications.

Large part of applications are standalone java applications, that read files, processes and insert, update, delete and query databases.

This is about 60% of my operations, and I would like to know and get information about their performance.

There's a way to make it with Elastic APM ? If yes how can I do it ? I have search but could not find.

targino.cbranco · December 29, 2021, 2:04pm

This is a image with my firsts applications working on Elastic APM

Sylvain_Juge · January 5, 2022, 4:52pm

Hi @targino.cbranco , I'm happy to hear that you have managed to validate the agent setup.

While the agent has automatic instrumentation for known libraries/APIs, this does not cover custom processing applications that might not rely on those.

In this case, there are multiple options described in our documentation.

The first thing to do here is to ensure that you can instrument your non-HTTP transactions in your application. Once this is done reading and writing from DB is usually done through standard APIs like JDBC that we already instrument.

system · January 26, 2022, 12:52pm

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Java Agent APM won't send data to Elasticsearch APM java , server	3	1109	February 9, 2023
Java Agent: No data has been received from agents yet APM java	7	1783	April 12, 2021
APM agent does not start sending any events APM java	2	149	April 19, 2024
APM "setup" shows success but no APM data available APM	27	2941	February 2, 2019
ELK - APM 7.0 - No data recevied from Java agent APM java	11	1960	June 11, 2019

Beginer error, sent data to APM but this is not showed on kibana

Related Topics