Apm server logs not telling if it's getting a request

Elastic Observability APM
,
1

Hi, using apm server on kubernetes. I have turned on the debug logs. The application guy is telling that he is sending the request to apm server with log message. How can I validate it in apm that it is processing that request? Apm server logs doesn't have any entry of request it received.

Also how can I validate that apm server is sending the request to elasticsearch. Apm logs doesn't have any info.

2

Hi @Divit_Sharma,
if the APM Server receives a request it generally logs something similar to <timestamp> INFO [request] middleware/log_middleware.go:101 request accepted <some details> on the INFO level.

In DEBUG mode it also logs information around how many events are published to elasticsearch. By default the server starts with log level INFO in which case nothing is logged for successfully sending data to Elasticsearch.

No log entries indicate that no requests are received by the APM Server. Since it sounds like the instrumented app contains logs for sending events to the server, is there a proxy between the app and the APM Server that might terminate/block the request?

3

Thanks for the information. Will revert tomorrow. No proxy is configured.

4

Hi,

I am able to telnet to apm server on kubernetes on 8200.

When I hit http://apmserver:8200, it showing me blank response. Absolutely nothing. No error as well.

In logs, I can see

Connection to backoff(elasticsearch ) established.

Successfully obtained connection to Kibana.

5

FYI, SECRET TOKEN had to be added. Changes should be made so that apm throws error that token is missing.

6

Glad to hear you found the issue.
When the secret token is configured and missing in a request, the APM Server usually logs an ERROR and the agent receives a response 401 unauthorized and is supposed to log information accordingly.

Could you provide more information about which APM Server version and agent you are using?

7

7.9.2 APM Server on kubernetes. DOTNET agent latest. Agent should not matter since even curl request without token gave blank response.

8

When calling http://apmserver:8200 without a secret token it is expected to return a 200 with an empty response body; whereas if a secret token were provided, detailed information about e.g. the server version would be returned. This is intended behavior, as it is used as a healthcheck endpoint that is also available for RUM agents (which do not support secret tokens).
However, when the agent makes a request to ingest data it calls http://apmserver:8200/intake/v2/events in which case a 401 is returned and an error is logged. In my previous comment I was referring to the agent requests.

If you'd like to try it out with curl, you can download a sample request body from the APM Server docs and run a command like curl -i -H "Content-type: application/x-ndjson" --data-binary @sample-data.ndjson http://localhost:8200/intake/v2/events

1 Like
9

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.