# APM "setup" shows success but no APM data available **URL:** \ **Category:** APM\ **Created:** [January 2, 2019, 11:42am UTC](https://discuss.elastic.co/t/apm-setup-shows-success-but-no-apm-data-available/162642 "2019-01-02T11:42:22Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![ethrbunny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethrbunny/32/34603_2.png) [@ethrbunny](https://discuss.elastic.co/u/ethrbunny)\ **Post date:** [January 2, 2019, 11:42am UTC](https://discuss.elastic.co/t/apm-setup-shows-success-but-no-apm-data-available/162642/1 "2019-01-02T11:42:22Z")
Attempting to send APM data from a Java app. Have followed "setup" instructions as closely as possible. On "setup instructions" page : - "APM Server status" shows "You have correctly setup APM server" - "Agent status" shows "Data successfully received from one or more agents" Going to "APM" page in Kibana shows "No services were found". My setup: - app running under kubernetes - apm server running as kubernetes daemonset App instance includes these settings: ``` JAVA_OPTS="-javaagent:elastic-apm-agent-1.1.0.jar -Delastic.apm.service_name=some_appname -Delastic.apm.application_packages=com.myapp.appname -Delastic.apm.server_url=http://127.0.0.1:8200" ``` ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [January 2, 2019, 12:21pm UTC](https://discuss.elastic.co/t/apm-setup-shows-success-but-no-apm-data-available/162642/2 "2019-01-02T12:21:29Z")
Hi and thanks for trying out our agent. Please run the query `GET apm-*/_search` on your Elasticsearch and see if and what data is stored. If there is data stored, please try to see if timestamps are as expected. ---
**Author:** ![ethrbunny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethrbunny/32/34603_2.png) [@ethrbunny](https://discuss.elastic.co/u/ethrbunny)\ **Post date:** [January 3, 2019, 11:02am UTC](https://discuss.elastic.co/t/apm-setup-shows-success-but-no-apm-data-available/162642/3 "2019-01-03T11:02:47Z")
There are indexes there. Looking at "management" "elasticsearch" "index management" it doesn't appear there is much data in there - if any. Similarly - going to "discover" in kibana doesn't return anything for those indexes. ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [January 3, 2019, 12:23pm UTC](https://discuss.elastic.co/t/apm-setup-shows-success-but-no-apm-data-available/162642/4 "2019-01-03T12:23:05Z")
Please try this query to see if there's any data from the Java agent: ```auto GET apm-*/_search { "query": { "term": { "context.service.agent.name": "java" } } } ``` You can use the "Dev Tools" UI for that in Kibana. If there is no data coming from the Java agent, there may a problem with the communication between agent and server. There is currently an issue about the setup steps showing connection is successful while it is not. Please try to look for interesting stuff in the agent and APM server logs and let us know if you find anything. Thanks, Eyal. ---
**Author:** ![ethrbunny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethrbunny/32/34603_2.png) [@ethrbunny](https://discuss.elastic.co/u/ethrbunny)\ **Post date:** [January 6, 2019, 1:54pm UTC](https://discuss.elastic.co/t/apm-setup-shows-success-but-no-apm-data-available/162642/5 "2019-01-06T13:54:20Z")
``` { "took" : 9, "timed_out" : false, "_shards" : { "total" : 30, "successful": 30, "skipped": 0, "failed": 0 }, "hits" : { "total" : 0, "max_score" : null, "hits" : [] } } ``` Looking for agent logs. ---
**Author:** ![digitalron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/digitalron/32/38224_2.png) [@digitalron](https://discuss.elastic.co/u/digitalron)\ **Post date:** [January 6, 2019, 4:14pm UTC](https://discuss.elastic.co/t/apm-setup-shows-success-but-no-apm-data-available/162642/6 "2019-01-06T16:14:17Z")
How's the connection from your agent to the APM Server? Can you get a response if you curl to http://[apm-server-address]:8200 coming from the process space where your agent is running on? ---
**Author:** ![ethrbunny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethrbunny/32/34603_2.png) [@ethrbunny](https://discuss.elastic.co/u/ethrbunny)\ **Post date:** [January 6, 2019, 6:12pm UTC](https://discuss.elastic.co/t/apm-setup-shows-success-but-no-apm-data-available/162642/7 "2019-01-06T18:12:38Z")
I can curl to the host ip:8200 and get a response. I updated the app to use that address for the **elastic.apm.server.url** address. Still no happiness from APM though. I reran the above GET but got the same response. ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [January 6, 2019, 6:31pm UTC](https://discuss.elastic.co/t/apm-setup-shows-success-but-no-apm-data-available/162642/8 "2019-01-06T18:31:20Z")
It should be `server_urls` with or without the prefix, depending on how you configure it. Please see [the related documentation](https://www.elastic.co/guide/en/apm/agent/java/current/config-reporter.html#config-server-urls). ---
**Author:** ![ethrbunny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethrbunny/32/34603_2.png) [@ethrbunny](https://discuss.elastic.co/u/ethrbunny)\ **Post date:** [January 6, 2019, 7:49pm UTC](https://discuss.elastic.co/t/apm-setup-shows-success-but-no-apm-data-available/162642/9 "2019-01-06T19:49:42Z")
I think **server\_urls** is just if you use a property file for the settings. Im passing them on the java cmd line. ---
**Author:** ![Eyal\_Koren](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eyal_koren/32/36830_2.png) [@Eyal\_Koren](https://discuss.elastic.co/u/Eyal_Koren)\ **Post date:** [January 6, 2019, 8:20pm UTC](https://discuss.elastic.co/t/apm-setup-shows-success-but-no-apm-data-available/162642/10 "2019-01-06T20:20:42Z")
Right, then `-Delastic.apm.server_urls`. Notice the `s` in the end, which I now see you omitted in the cmd line you enclosed at the beginning ---
**Author:** ![ethrbunny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethrbunny/32/34603_2.png) [@ethrbunny](https://discuss.elastic.co/u/ethrbunny)\ **Post date:** [January 7, 2019, 11:09am UTC](https://discuss.elastic.co/t/apm-setup-shows-success-but-no-apm-data-available/162642/11 "2019-01-07T11:09:39Z")
Dang. Unfortunately that didn't change anything. And FWIW -\> the 'setup instructions' page does have the param as **-Delastic.apm.server\_url=http://ip:8200** --\> definitely no 's'. But I did try it both with and without the extra letter. Is there some way to test whether or not anything is being sent? And on the APM page - do I leave the 'search' line (with the hourglass) empty? Asterisk? other? ---
**Author:** ![nugusbayevkk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nugusbayevkk/32/126683_2.png) [@nugusbayevkk](https://discuss.elastic.co/u/nugusbayevkk)\ **Post date:** [January 7, 2019, 11:18am UTC](https://discuss.elastic.co/t/apm-setup-shows-success-but-no-apm-data-available/162642/12 "2019-01-07T11:18:21Z")
hi, can you share your application's startup logs? ---
**Author:** ![ethrbunny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethrbunny/32/34603_2.png) [@ethrbunny](https://discuss.elastic.co/u/ethrbunny)\ **Post date:** [January 7, 2019, 11:28am UTC](https://discuss.elastic.co/t/apm-setup-shows-success-but-no-apm-data-available/162642/13 "2019-01-07T11:28:41Z")
Im thinking that my simple java app isn't sending anything for APM to pick up. Looking at the startup logs I don't see any reference to APM. ``` [root@playground kafka_lagmonitor]# kubectl --namespace=kafka logs kafka-lagmonitor-558bdbdf9c-dck9g 11:21:55,942 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [./logback.xml] 11:21:55,942 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback.groovy] 11:21:55,942 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Could NOT find resource [logback-test.xml] 11:21:55,942 |-INFO in ch.qos.logback.classic.LoggerContext[default] - Found resource [logback.xml] at [jar:file:/kafka_lagmonitor-all.jar!/logback.xml] 11:21:55,960 |-INFO in ch.qos.logback.core.joran.spi.ConfigurationWatchList@6842775d - URL [jar:file:/kafka_lagmonitor-all.jar!/logback.xml] is not of type file 11:21:56,019 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - debug attribute not set 11:21:56,025 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.ConsoleAppender] 11:21:56,031 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [STDOUT] 11:21:56,038 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property 11:21:56,095 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.FileAppender] 11:21:56,097 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [FILE] 11:21:56,098 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter@11:83 - no applicable action for [rollingPolicy], current ElementPath is [[configuration][appender][rollingPolicy]] 11:21:56,098 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter@12:30 - no applicable action for [fileNamePattern], current ElementPath is [[configuration][appender][rollingPolicy][fileNamePattern]] 11:21:56,098 |-ERROR in ch.qos.logback.core.joran.spi.Interpreter@13:25 - no applicable action for [maxHistory], current ElementPath is [[configuration][appender][rollingPolicy][maxHistory]] 11:21:56,099 |-INFO in ch.qos.logback.core.joran.action.NestedComplexPropertyIA - Assuming default type [ch.qos.logback.classic.encoder.PatternLayoutEncoder] for [encoder] property 11:21:56,104 |-INFO in ch.qos.logback.core.FileAppender[FILE] - File property is set to [/var/log/lagmonitor.log] 11:21:56,106 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.http] to WARN 11:21:56,106 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[org.apache.http] 11:21:56,107 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [net.sf] to WARN 11:21:56,107 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[net.sf] 11:21:56,107 |-INFO in ch.qos.logback.classic.joran.action.LoggerAction - Setting level of logger [org.apache.kafka] to WARN 11:21:56,107 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[org.apache.kafka] 11:21:56,107 |-INFO in ch.qos.logback.classic.joran.action.RootLoggerAction - Setting level of ROOT logger to INFO 11:21:56,107 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [STDOUT] to Logger[ROOT] 11:21:56,107 |-INFO in ch.qos.logback.classic.joran.action.ConfigurationAction - End of configuration. 11:21:56,110 |-INFO in ch.qos.logback.classic.joran.JoranConfigurator@574caa3f - Registering current configuration as safe fallback point ``` After that it's just app data. NOTE - the apm docs (linked in earlier message) and the "apm setup" page are conflicting. One lists the param with the 's' and the other omits it. ---
**Author:** ![nugusbayevkk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nugusbayevkk/32/126683_2.png) [@nugusbayevkk](https://discuss.elastic.co/u/nugusbayevkk)\ **Post date:** [January 7, 2019, 11:34am UTC](https://discuss.elastic.co/t/apm-setup-shows-success-but-no-apm-data-available/162642/14 "2019-01-07T11:34:53Z")
can you enter in your pod and check that javaagent\*\*.jar is exist? can you share pod.yml configuration here and docker image creating commands? ---
**Author:** ![ethrbunny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethrbunny/32/34603_2.png) [@ethrbunny](https://discuss.elastic.co/u/ethrbunny)\ **Post date:** [January 7, 2019, 11:44am UTC](https://discuss.elastic.co/t/apm-setup-shows-success-but-no-apm-data-available/162642/15 "2019-01-07T11:44:08Z")
``` [root@kafka-lagmonitor-558bdbdf9c-55b8t /]# ls -al elastic* ``` total 42588 -rw-r--r-- 1 root root 7233670 Nov 27 13:55 elastic-apm-agent-1.1.0.jar Docker command: ``` docker build -t docker.io/ethrbunny/kafka_lagmonitor . ``` K8 config: ``` [root@playground kafka_lagmonitor]# cat lagmonitor_k8.yml apiVersion: apps/v1 kind: Deployment metadata: namespace: kafka name: kafka-lagmonitor labels: app: kafka-lagmonitor spec: selector: matchLabels: app: kafka-lagmonitor template: metadata: labels: app: kafka-lagmonitor spec: containers: - name: kafka-lagmonitor image: docker.io/ethrbunny/kafka_lagmonitor:latest dnsPolicy: "None" dnsConfig: nameservers: - 10.95.96.103 ``` ---
**Author:** ![nugusbayevkk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nugusbayevkk/32/126683_2.png) [@nugusbayevkk](https://discuss.elastic.co/u/nugusbayevkk)\ **Post date:** [January 7, 2019, 11:46am UTC](https://discuss.elastic.co/t/apm-setup-shows-success-but-no-apm-data-available/162642/16 "2019-01-07T11:46:13Z")
@ethrbunny , add please your Dockerfile. ---
**Author:** ![digitalron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/digitalron/32/38224_2.png) [@digitalron](https://discuss.elastic.co/u/digitalron)\ **Post date:** [January 7, 2019, 1:16pm UTC](https://discuss.elastic.co/t/apm-setup-shows-success-but-no-apm-data-available/162642/17 "2019-01-07T13:16:33Z")
Can you try sending a request from your service to the apm-server at http://[apm-server-url]:8200/v1/transactions using the payload below: (you need to add a header Content-Type: application/json) ``` { "service": { "name": "mock_service", "agent": { "name": "mock_agent", "version": "1.0" } }, "transactions": [ { "id": "945254c5-67a5-417e-8a4e-aa29efcc22e3", "type": "request", "duration": 15.000000 } ] } ``` If the connection between apm-server and elastic and between agent and apm-server is good, you should get an entry **mock\_service** in the APM view. ---
**Author:** ![ethrbunny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethrbunny/32/34603_2.png) [@ethrbunny](https://discuss.elastic.co/u/ethrbunny)\ **Post date:** [January 7, 2019, 2:39pm UTC](https://discuss.elastic.co/t/apm-setup-shows-success-but-no-apm-data-available/162642/18 "2019-01-07T14:39:01Z")
Got some activity from the **mock\_agent** POST. I used **GET apm-\*/\_service** to check. ---
**Author:** ![ethrbunny](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ethrbunny/32/34603_2.png) [@ethrbunny](https://discuss.elastic.co/u/ethrbunny)\ **Post date:** [January 7, 2019, 2:40pm UTC](https://discuss.elastic.co/t/apm-setup-shows-success-but-no-apm-data-available/162642/19 "2019-01-07T14:40:27Z")
Dockerfile: ``` FROM docker.io/ethrbunny/centos-java:latest COPY build/libs/kafka_lagmonitor-all.jar / COPY elastic-apm-agent-1.1.0.jar / ENV JAVA_OPTS="-javaagent:elastic-apm-agent-1.1.0.jar -Delastic.apm.service_name=kafka_lagmonitor -Delastic.apm.application_packages=com.cedexis.lagmonitor -Delastic.apm.server_urls=http://10.95.96.144:8200" CMD java -Dlogback.configurationFile=./logback.xml $JAVA_OPTS -jar /kafka_lagmonitor-all.jar ``` ---
**Author:** ![nugusbayevkk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nugusbayevkk/32/126683_2.png) [@nugusbayevkk](https://discuss.elastic.co/u/nugusbayevkk)\ **Post date:** [January 7, 2019, 2:47pm UTC](https://discuss.elastic.co/t/apm-setup-shows-success-but-no-apm-data-available/162642/20 "2019-01-07T14:47:33Z")
@ethrbunny what does the command `echo $JAVA_OPTS` return inside that pod? [Next page](https://discuss.elastic.co/t/apm-setup-shows-success-but-no-apm-data-available/162642.md?page=2)