APM on docker did not send logs trace to elastic cloud

bekk777 · January 27, 2022, 6:37pm

Hello everyone,
I don´t understand why my configuration APM agent and server working on docker. But on the elastic cloud does not show anything on the APM service. But in Discover metrics show.
cloud version v 7.15.2.
Can someone please explain what I should do first? I'm new to elastic and little understand everything.
This is like my problem.
https://discuss.elastic.co/t/how-to-use-apm-agent-for-hybris-application/164921

Thanks, advanced

Sylvain_Juge · February 1, 2022, 8:22am

Hi @bekk777 , welcome to the forum.

The first thing to check is your agent setup, can you describe what you have done to add the agent so we can help you on the missing parts ?

For example, do you use the -javaagent JVM option, or rely on runtime attachment (either through API or through CLI ?). Do you see anything in your application logs and/or standard output ?

bekk777 · February 1, 2022, 9:44am

Hi Sylvain,
Thank you answer. Yes sure
This is my APM-Server which is running on docker.

apm-server:
        host: "0.0.0.0:8200"
output.elasticsearch:
        hosts: ["ip-elasticsearch:9200"]
        username: "elastic"
        password: "password"

Java agent configures like this manual: Programmatic API setup to self-attach | APM Java Agent Reference [1.x] | Elastic
Both of them are in docker run (APM-Server and java agent on the same network)
Logs from APM-server:

2022-02-01T09:30:04.762Z        DEBUG   [elasticsearch] elasticsearch/client.go:719     Ping request failed with: Get http://ip-elasticsearch:9200: dial tcp ip-elasticsearch:9200: connect: connection refused
2022-02-01T09:30:20.649Z        INFO    [request]       beater/common_handlers.go:272   handled request {"request_id": "85021106-a8cd-4d4c-9d4e-012bfecef9cf", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "docker-ip", "user-agent": "apm-agent-java/1.28.4 (test-agent-be)", "response_code": 202}
2022-02-01T09:30:20.650Z        DEBUG   [publish]       pipeline/processor.go:309       Publish event: {
  "@timestamp": "2022-02-01T09:30:10.654Z",
  "@metadata": {
    "beat": "apm-server",
    "type": "doc",
    "version": "6.8.11"

I am something wrong, I am pretty sure. But 4 days I am working do everything I can
The last hope is for you.
Thank advance

Sylvain_Juge · February 1, 2022, 10:26am

I'm not familiar with your Docker setup, but it looks like a network issue, are you using Docker compose or similar ?

ip-elasticsearch should be the DNS/IP name of the ES server and should be accessible from APM server container.

The apm-server container should be accessible from the application running with the agent. If your application needs to access a database backend the configuration should be similar here.

bekk777 · February 1, 2022, 11:07am

Thank you for replaying.
I will check that, and send here feedback.

bekk777 · February 1, 2022, 1:09pm

On docker:

docker run --name=apm-server --network apm --dit  docker.elastic.co/apm/apm-server:7.15.2 -p 8200:8200

And java agent was configured to listen apm-server:8200.

Sylvain_Juge · February 2, 2022, 9:04am

The simplest solution to check for connection issues between the apm-agent and apm-server is to use the "health check" API endpoint on APM server with curl.

From the container that runs the application, you should use one of the following curl commands to check for connection issues. If this succeeds you should get an HTTP 200 response with JSON body.

For those examples, I'm assuming the name of apm-server is apm-server and that it's using plain http, not https.

# Without authentication
curl --request GET \
  --url http://apm-server:8200/

# When using secret token
curl --request GET \
  --url http://apm-server:8200/ \
  --header 'Authorization: Bearer <secret_token>'
 
# When using API key
curl --request GET \
  --url http://apm-server:8200/ \
  --header 'Authorization: ApiKey <api_key>'

bekk777 · February 3, 2022, 2:11pm

Hi Sylvain.
Thank you for your help.

When I checked on the app server (which is running on docker and java agent there too), I got this:

{"ok":{"build_date":"2020-07-09T18:01:36Z","build_sha":"dda44e48cd9c3bac7e5a359a14625284d1d10347","version":"6.8.11"}}root@ccd045e291db:

So, APM agent running correctly and request send from APM agent to APM server er correctly, right?
But when I am sending from the APM server to the elastic server, I got this:

curl: (7) Failed connect to public-IP-elastic:8200; Connection refused

Request was: curl public-IP-elastic:8200 -d '{"username": "elastic", "password": "secret_token"}' -v

Something wrong between the APM server and elastic, right?

Regards,
Nasimjon.

bekk777 · February 3, 2022, 2:24pm

In addition, metrics are coming to the elastic server. I saw in discovery part on Kibana. It is a surprise for me.

Sylvain_Juge · February 3, 2022, 3:23pm

The URL/port of elastic server are not the same between apm agent and apm-server and between apm-server and elastic.

Can we conclude that it's now working as expected since you now have metrics ?

bekk777 · February 3, 2022, 8:29pm

No The metrics came right away when I was setting up the APM server :). But there is nothing in APM in services and transactions. What I wanna expecting. I am expecting this kind of service:
DEMO APM Services

I believe that you are helping me.
Thanks.

Sylvain_Juge · February 7, 2022, 8:06am

The agent collects passively the application activity, unlike metrics which are collected through a scheduled task. I'm assuming that the agent setup works as expected since you are able to get metrics.

When you say "metrics are coming to the elastic server", can you describe in which index are they stored in ? Do you have a few sample documents ? There might be some system-metrics related metrics captured, and those are not the same as metrics captured through APM.

If you don't see transactions in Kibana, it means likely one of the following situations:

the application has no activity, triggering a few transactions on the application is required
the application relies on a technology that is not supported out of the box, checking supported technologies

bekk777 · February 15, 2022, 9:12pm

Hey Sylvain.
Sorry, I didn't reply for a long time.
This is a screenshot.

On APM-* index is stored.

bekk777 · February 20, 2022, 7:05pm

Hey Sylvain,
One question:
Can I send APM-service not directly to ELK server, via Logstash, and maybe another solution?
Thanks in advance.

Jack_Shirazi · February 22, 2022, 1:37pm

Not sure what you are asking. The Java agent sends to an APM server (the URL you set) which then sends to Elasticsearch, you view the data that is held in Elasticsearch from kibana. The logs are not sent by the agent, and you need a different mechanism, like logstash, to have those sent

bekk777 · February 22, 2022, 2:30pm

Hey Jack,
Thanks for replying.
Yes, because when I send it to ELK-server, not show me anything on services APM, transaction. I think maybe it will be possible for the first to handle it (to Logstash from APM-server) and then send it to the ELK server.
This is a diagram of my APM server.

How can I configure like DEMO link, which is mentioned above?

Jack_Shirazi · February 23, 2022, 1:56pm

Can we clarify exactly what we have and what is missing.

You have a Java running application with a Java agent which is self-attaching (I assume you modified your main class to do this). I assume that the Java application is showing correct functionality. The Java application is running in a docker container
You have an APM agent running in a (different? the same?) docker container on the default 8200 port. The Java agent is configured to point to the APM agent on localhost 8200.
You can curl successfully (from inside the docker container?) to the APM agent (curl http://localhost:8200 ?) and the APM agent reports it is running version 6.8.11 (which is a very old version)
There are metrics being received by the Elasticsearch node, including one example metric which includes garbage collector name and heap stats (I assume this is from the Java application, ie that there is no other Java application running against the system with the given hostname). But this is a beat metric, not a Java agent metric.
You are expecting to see a transaction but you are not seeing that
Please correct anything that is incorrect.

Have you turned debug log level on for the agent to see if the Java agent is connecting and instrumenting what you expect?

Exactly what transaction are you expecting to see? If your Java application is some custom classes which execute custom functionality, the Java agent does not know what to call a transaction. The out-of-the-box technologies the agent will instrument are Supported technologies | APM Java Agent Reference [1.x] | Elastic - otherwise the agent needs you to explicitly specify the transactions

bekk777 · February 23, 2022, 3:07pm

Hey Jack.
Thanks for your help.
Yes, you are absolutely correct. But one thing to be more clear: Docker is running two containers that have a single network (same network). The Java agent is configured to listen on the network with the APM-server port, for example, in my case, apm-server:8200. And the APM server sends further to the ElK-server.
I am not sure about "debug log level". Today I'll take a look and give you an answer to this.
In addition, if I do all these procedures from my PC, I send it to the elk server, everything works for me (Example). What I want. But with docker, no.

bekk777 · February 24, 2022, 4:48pm

From java agent i got this: ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type JSON_WRITER with this error: Connection refused
From apm server i got this: 2022-02-24T16:38:07.428Z INFO pipeline/output.go:105 Connection to backoff(elasticsearch(http://elasticsearch-host:9200)) established 2022-02-24T16:38:37.766Z INFO [request] beater/common_handlers.go:272 handled request {"request_id": "ca9cc236-7527-4899-8bd5-c7237ddf0f2c", "method": "POST", "URL": "/intake/v2/events", "content_length": -1, "remote_address": "172.19.0.2", "user-agent": "apm-agent-java/1.28.4 (java-app)", "response_code": 202}

Jack_Shirazi · February 24, 2022, 5:37pm

So that says the Java agent is NOT connecting to the APM agent in the same docker - you have some kind of inter-container connection problem, either the port is mapped wrong or the network is not the same. This is consistent with the rest of what you see - the APM agent has connected to Elasticsearch and has sent data (that beat metric), but the Java agent has not connected to the APM agent so you see nothing from it.
I can't help further here, this is docker specific and depends on how you have setup your containers