APM do not work

Elastic Observability APM
1

I'm trying to work with APM agent and server.
Tryinf to monitoring an java application that run under Tomcat 8.5 (windows)
I have installed the APM server on same host of ELK stack is installed. This is a test enviroment.
First question is that if I check the service with command
sudo service apm-server status

It says that service is active, but into the folder of the APM (/usr/share/apm-server nor /etc/apm-server) I cn't see any log file of apm-server. This is the 1st strange thing.
The second one is that if I PUSH, on KIBANA UI - the buttons:
"check agent status" or "check APM server status" these always report an OK answer, even if the server is stopped....

I have updated the ELK stack to latest version 6.5 and also the agent is at latest version - for java is elastic-apm-agent-1.0.1.jar

At agent side I have created a setenv.bat into the bin folder of Tomcat with there content:

set CATALINA_OPTS=%CATALINA_OPTS% -javaagent:C:\Program\apm-agent\elastic-apm-agent-1.0.1.jar
set CATALINA_OPTS=%CATALINA_OPTS% -Delastic.apm.service_name=my-app-name
set CATALINA_OPTS=%CATALINA_OPTS% -Delastic.apm.application_packages=mylabo.local
set CATALINA_OPTS=%CATALINA_OPTS% -Delastic.apm.server_urls=http://172.30.6.64:8200

Even the Agent do not create any log...
What can be the problem?

From the agent , when APM server is runnint I can browse tcp port 8200 and this reports:
{"ok":{"build_date":"2018-11-09T17:48:30Z","build_sha":"2c46a2919dbe06767caeacbffd0055aab1493991","version":"6.5.0"}}

2

It says that service is active, but into the folder of the APM (/usr/share/apm-server nor /etc/apm-server) I cn't see any log file of apm-server. This is the 1st strange thing.

The APM Server logs to /var/log/apm-server by default.

(I can't answer the Tomcat/Java bits -- I'll leave that to someone else.)

3

The agent logs to System.out by default. IIRC, that goes to catalina.log in case of tomcat. You can also specify the log_file option to let the agent log into its own file.

The OK answer just means that there are documents from the agent and the APM server in the index.

Just to be sure - can you see your application when you click on the APM tab of Kibana?

If not, the agent and server logs would be the next thing to diagnose.

4

SERVER SIDE.
Logs says that apm-server is running:

tail -f /var/log/apm-server/apm-server
2018-11-20T08:38:06.939Z        INFO    [server]        beater/server.go:67     Listening on: 172.30.6.64:8200
2018-11-20T08:38:06.939Z        INFO    [server]        beater/server.go:72     RUM endpoints disabled
2018-11-20T08:38:06.967Z        WARN    [handler]       beater/common_handlers.go:212   '/healthcheck' endpoint is deprecated, please use '/' instead
2018-11-20T08:38:06.967Z        INFO    [request]       beater/common_handlers.go:272   handled request {"request_id": "c7b1bdd3-dbc7-43f5-9b8c-047a92d817e2", "method": "GET", "URL": "/healthcheck", "content_length": 0, "remote_address": "172.30.6.64", "user-agent": "Go-http-client/1.1", "response_code": 200}
2018-11-20T08:38:06.968Z        INFO    [http_client]   beater/client.go:49     HTTP Server ready
2018-11-20T08:38:06.968Z        INFO    [onboarding]    beater/onboarding.go:36 Publishing onboarding document
2018-11-20T08:38:07.973Z        INFO    pipeline/output.go:95   Connecting to backoff(elasticsearch(http://172.30.6.64:9200))
2018-11-20T08:38:07.997Z        INFO    elasticsearch/client.go:712     Connected to Elasticsearch version 6.5.0
2018-11-20T08:38:07.998Z        INFO    template/load.go:129    Template already exists and will not be overwritten.
2018-11-20T08:38:07.998Z        INFO    pipeline/output.go:105  Connection to backoff(elasticsearch(http://172.30.6.64:9200)) established

I think the problem is at AGENT-SIDE.
Here as I told before , there is a Tomcat 8.5 on Windows server. I have created the setenv.bat.
I have also created a file called "elasticapm.properties" (in tha same folder of agent -jar file)
with this content:
service_name=my-app-name
application_packages=labo.local
server_urls=http://172.30.6.64:8200
But seems that Tomcat do not take care of this parameters at startup...
How can I verify if the agent is working right ?
Where are the agent log ?
REgards
P.Preformatted text

5

In the catalina.log I found this message:

20-Nov-2018 11:06:59.685 WARNING [localhost-startStop-2] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [leasing] appears to have started a thread named [elasticsearch[client][[timer]]] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
java.lang.Thread.sleep(Native Method)
org.elasticsearch.threadpool.ThreadPool$CachedTimeThread.run(ThreadPool.java:536)

Any idea ?

6

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.