APM "setup" shows success but no APM data available

Elastic Observability APM
1

Attempting to send APM data from a Java app. Have followed "setup" instructions as closely as possible.

On "setup instructions" page :

  • "APM Server status" shows "You have correctly setup APM server"
  • "Agent status" shows "Data successfully received from one or more agents"

Going to "APM" page in Kibana shows "No services were found".

My setup:

  • app running under kubernetes
  • apm server running as kubernetes daemonset

App instance includes these settings:

JAVA_OPTS="-javaagent:elastic-apm-agent-1.1.0.jar -Delastic.apm.service_name=some_appname  -Delastic.apm.application_packages=com.myapp.appname -Delastic.apm.server_url=http://127.0.0.1:8200"
2

Hi and thanks for trying out our agent.
Please run the query GET apm-*/_search on your Elasticsearch and see if and what data is stored. If there is data stored, please try to see if timestamps are as expected.

3

There are indexes there. Looking at "management" "elasticsearch" "index management" it doesn't appear there is much data in there - if any. Similarly - going to "discover" in kibana doesn't return anything for those indexes.

4

Please try this query to see if there's any data from the Java agent:

GET apm-*/_search
{
  "query": {
    "term": {
      "context.service.agent.name": "java"
    }
  }
}

You can use the "Dev Tools" UI for that in Kibana.

If there is no data coming from the Java agent, there may a problem with the communication between agent and server. There is currently an issue about the setup steps showing connection is successful while it is not. Please try to look for interesting stuff in the agent and APM server logs and let us know if you find anything.

Thanks,
Eyal.

5 
{
    "took" : 9,
    "timed_out" : false,
    "_shards" : {
        "total" : 30,
        "successful": 30,
        "skipped": 0,
        "failed": 0
     },
     "hits" : {
        "total" : 0,
        "max_score" : null,
       "hits" : []
  }
}

Looking for agent logs.

6

How's the connection from your agent to the APM Server? Can you get a response if you curl to http://[apm-server-address]:8200 coming from the process space where your agent is running on?

7

I can curl to the host ip:8200 and get a response. I updated the app to use that address for the elastic.apm.server.url address.

Still no happiness from APM though. I reran the above GET but got the same response.

8

It should be server_urls with or without the prefix, depending on how you configure it. Please see the related documentation.

9

I think server_urls is just if you use a property file for the settings. Im passing them on the java cmd line.

10

Right, then -Delastic.apm.server_urls. Notice the s in the end, which I now see you omitted in the cmd line you enclosed at the beginning

11

Dang. Unfortunately that didn't change anything.

And FWIW -> the 'setup instructions' page does have the param as -Delastic.apm.server_url=http://ip:8200 --> definitely no 's'.

But I did try it both with and without the extra letter.

Is there some way to test whether or not anything is being sent? And on the APM page - do I leave the 'search' line (with the hourglass) empty? Asterisk? other?

12

hi, can you share your application's startup logs?

13

Im thinking that my simple java app isn't sending anything for APM to pick up.

Looking at the startup logs I don't see any reference to APM.

[root@playground kafka_lagmonitor]# kubectl --namespace=kafka logs kafka-lagmonitor-558bdbdf9c-dck9g 
11:21:55,942 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [./logback.xml]
11:21:55,942 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy]
11:21:55,942 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml]
11:21:55,942 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [jar:file:/kafka_lagmonitor-all.jar!/logback.xml]
11:21:55,960 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@6842775d - URL [jar:file:/kafka_lagmonitor-all.jar!/logback.xml] is not of type file
11:21:56,019 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set
11:21:56,025 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender]
11:21:56,031 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT]
11:21:56,038 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
11:21:56,095 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.FileAppender]
11:21:56,097 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [FILE]
11:21:56,098 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter@11:83 - no applicable action for [rollingPolicy], current ElementPath is [[configuration][appender][rollingPolicy]]
11:21:56,098 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter@12:30 - no applicable action for [fileNamePattern], current ElementPath is [[configuration][appender][rollingPolicy][fileNamePattern]]
11:21:56,098 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter@13:25 - no applicable action for [maxHistory], current ElementPath is [[configuration][appender][rollingPolicy][maxHistory]]
11:21:56,099 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property
11:21:56,104 |-INFO in ch.qos.logback.core.FileAppender[FILE] - File property is set to [/var/log/lagmonitor.log]
11:21:56,106 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.http] to WARN
11:21:56,106 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[org.apache.http]
11:21:56,107 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [net.sf] to WARN
11:21:56,107 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[net.sf]
11:21:56,107 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.kafka] to WARN
11:21:56,107 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[org.apache.kafka]
11:21:56,107 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO
11:21:56,107 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT]
11:21:56,107 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration.
11:21:56,110 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@574caa3f - Registering current configuration as safe fallback point

After that it's just app data.

NOTE - the apm docs (linked in earlier message) and the "apm setup" page are conflicting. One lists the param with the 's' and the other omits it.

14

can you enter in your pod and check that javaagent**.jar is exist?
can you share pod.yml configuration here and docker image creating commands?

15 
[root@kafka-lagmonitor-558bdbdf9c-55b8t /]# ls -al elastic*

total 42588
-rw-r--r-- 1 root root 7233670 Nov 27 13:55 elastic-apm-agent-1.1.0.jar

Docker command:

docker build -t docker.io/ethrbunny/kafka_lagmonitor .

K8 config:

[root@playground kafka_lagmonitor]# cat lagmonitor_k8.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: kafka
  name: kafka-lagmonitor
  labels: 
    app: kafka-lagmonitor
spec:
  selector:
    matchLabels:
      app: kafka-lagmonitor
  template:
    metadata:
      labels:
        app: kafka-lagmonitor
     spec:
      containers:
      - name: kafka-lagmonitor
         image: docker.io/ethrbunny/kafka_lagmonitor:latest
      dnsPolicy: "None"
      dnsConfig:
        nameservers:
          - 10.95.96.103
16

@ethrbunny , add please your Dockerfile.

17

Can you try sending a request from your service to the apm-server at http://[apm-server-url]:8200/v1/transactions using the payload below:
(you need to add a header Content-Type: application/json)

{
    "service": {
        "name": "mock_service",
        "agent": {
            "name": "mock_agent",
            "version": "1.0"
        }
    },
    "transactions": [
        {
            "id": "945254c5-67a5-417e-8a4e-aa29efcc22e3",
            "type": "request",
            "duration": 15.000000
        }
    ]
}

If the connection between apm-server and elastic and between agent and apm-server is good, you should get an entry mock_service in the APM view.

18

Got some activity from the mock_agent POST.

I used GET apm-*/_service to check.

19

Dockerfile:

FROM docker.io/ethrbunny/centos-java:latest
COPY build/libs/kafka_lagmonitor-all.jar /
COPY elastic-apm-agent-1.1.0.jar /

ENV JAVA_OPTS="-javaagent:elastic-apm-agent-1.1.0.jar -Delastic.apm.service_name=kafka_lagmonitor  -Delastic.apm.application_packages=com.cedexis.lagmonitor -Delastic.apm.server_urls=http://10.95.96.144:8200"

CMD java -Dlogback.configurationFile=./logback.xml $JAVA_OPTS  -jar /kafka_lagmonitor-all.jar
20

@ethrbunny what does the command echo $JAVA_OPTS return inside that pod?