APM agent showing timeout Error (Connection Refused)

Kibana version: 6.6.0

Elasticsearch version: 6.6.0

APM Server version: 6.6.0

APM Agent language and version:

Browser version: N/A

Original install method:
Java using the agent jar

Fresh install or upgraded from other version?
Fresh install

Is there anything special in your setup?

No special setup as far as I am aware, just followed the guide on setting up the APM agent. I am using the APM server in the cloud option, and have the obtained the server URL and secret token from there.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):

When setting up the APM Java agent in the docker container that I have running, the agent thows an error saying:

ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: Connection refused (Connection refused)

I have included the connection information in the elasticapm.properties file, as can be seen below:

############################################
# Core                                     #
############################################

Type: Boolean
Default value: true
active=true

Type: Boolean
Default value: true
instrument=true

# The name of your service
# This setting can not be changed at runtime. Changes require a restart of the application.
Type: String
Default value: Main
service_name=Main

# A version string for the currently deployed version of the service. If you don’t version your deployments, the recommended value for this field is the commit identifier of the deployed revision, e.g. the output of git rev-parse HEAD.
#
# This setting can not be changed at runtime. Changes require a restart of the application.
# Type: String
# Default value:
#
service_version= 1.0

# The name of the environment this service is deployed in, e.g. "production" or "staging".
Type: String
#Default value:
environment=test

# By default, the agent will sample every transaction (e.g. request to your service). To reduce overhead and storage requirements, you can set the sample rate to a value between 0.0 and 1.0. We still record overall time and the result for unsampled transactions, but no context information, tags, or spans.
Type: Double
#Default value: 1.0
transaction_sample_rate=1.0

# Limits the amount of spans that are recorded per transaction.
# This is helpful in cases where a transaction creates a very high amount of spans (e.g. thousands of SQL queries).
# Setting an upper limit will prevent overloading the agent and the APM server with too much work for such edge cases.
# This setting can be changed at runtime
Type: Integer
#Default value: 500
transaction_max_spans=500


Type: Level
#Default value: INFO
log_level=DEBUG

#This setting can not be changed at runtime. Changes require a restart of the application.
Type: comma separated list
Default value: https://7aea994c47fc41749d1faa5a9e48b604.apm.us-east-1.aws.cloud.es.io:443
server_urls= https://7aea994c47fc41749d1faa5a9e48b604.apm.us-east-1.aws.cloud.es.io:443
# This setting can not be changed at runtime. Changes require a restart of the application.
# Type: String
# Default value:
secret_token=XXXXXXXXXXXXXXXX

Summary
I am not sure what is missing that is causing the connection to be refused, but imagine that perhaps there is a step that I am missing. Any assistance would be greatly appreciated.

Hi and thanks for using the Java agent!

Could you provide the startup logs? See also https://www.elastic.co/guide/en/apm/agent/java/current/trouble-shooting.html#trouble-shooting-logging

Thanks,
Felix

Hi Felix!

Thanks for the quick response. Here is the startup log. Please note that since this is running on a docker container there are multiple services and logs are not all related to the agent. I have removed most of what is not related, but somethings I was unsure about, so I left them in just in case.

2019-03-11T17:50:45.498Z        INFO    instance/beat.go:544    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2019-03-11T17:50:45.510Z        INFO    instance/beat.go:551    Beat UUID: e82c80d9-a384-47a2-98ad-0efa30d6675e
2019-03-11T17:50:45.510Z        INFO    [beat]  instance/beat.go:768    Beat info       {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "e82c80d9-a384-47a2-98ad-0efa30d6675e"}}}
2019-03-11T17:50:45.510Z        INFO    [beat]  instance/beat.go:777    Build info      {"system_info": {"build": {"commit": "e193f6d68b25b7ddbe3a3ed8d60bc07fea1ef800", "libbeat": "6.4.2", "time": "2018-09-26T12:42:46.000Z", "version": "6.4.2"}}}
2019-03-11T17:50:45.510Z        INFO    [beat]  instance/beat.go:780    Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":4,"version":"go1.10.3"}}}
2019-03-11T17:50:45.511Z        INFO    [beat]  instance/beat.go:784    Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2019-02-25T20:21:13Z","containerized":true,"hostname":"ae1c69cf3ebf","ips":["127.0.0.1/8","172.19.0.8/24"],"kernel_version":"4.15.0-1037-azure","mac_addresses":["02:42:ac:13:00:08"],"os":{"family":"debian","platform":"debian","name":"Debian GNU/Linux","version":"","major":0,"minor":0,"patch":0},"timezone":"UTC","timezone_offset_sec":0,"id":"61416584cb417f53bdd8abd092d38401"}}}
2019-03-11T17:50:45.511Z        INFO    [beat]  instance/beat.go:813    Process info    {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null}, "cwd": "/", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 18, "ppid": 17, "seccomp": {"mode":"filter","no_new_privs":false}, "start_time": "2019-03-11T17:50:44.850Z"}}}
2019-03-11T17:50:45.511Z        INFO    instance/beat.go:273    Setup Beat: filebeat; Version: 6.4.2
2019-03-11T17:50:45.511Z        INFO    elasticsearch/client.go:163     Elasticsearch url: https://90217d99e4a844d5b102a5ef2354ce76.us-east-1.aws.found.io:9243
2019-03-11T17:50:45.512Z        INFO    pipeline/module.go:98   Beat name: ae1c69cf3ebf
[ ok ] Starting nginx: nginx.
2019-03-11 17:50:46.208 [apm-server-healthcheck] WARN co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is not available (Connection refused (Connection refused))
2019-03-11 17:50:46.224 [Attach Listener] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.4.0 as Main on Java 10.0.2 (Oracle Corporation) Linux 4.15.0-1037-azure
2019-03-11 17:50:46.225 [Attach Listener] WARN co.elastic.apm.agent.configuration.StartupInfo - To enable all features and to increase startup times, please configure application_packages
2019-03-11 17:50:49 INFO Logging initialized @4396ms to org.eclipse.jetty.util.log.Slf4jLog
2019-03-11 17:50:50 INFO jetty-9.4.12.v20180830; built: 2018-08-30T13:59:14.071Z; git: 27208684755d94a92186989f695db2d7b21ebc51; jvm 10.0.2+13-Debian-2
2019-03-11 17:50:50 INFO DefaultSessionIdManager workerName=node0
2019-03-11 17:50:50 INFO No SessionScavenger set, using defaults
2019-03-11 17:50:50 INFO node0 Scavenging every 660000ms
2019-03-11 17:50:50 INFO Started i.j.c.u.@3642aeeb{/,null,AVAILABLE}
2019-03-11 17:50:50 INFO Started o.e.j.s.ServletContextHandler@f008331{/,null,AVAILABLE}
2019-03-11 17:50:50 INFO Started ServerConnector@7f1a22e9{HTTP/1.1,[http/1.1]}{0.0.0.0:8081}
2019-03-11 17:50:50 INFO Started @4683ms
2019-03-11 17:50:50 INFO Jetty is listening on: [http://localhost:8081]
2019-03-11 17:51:16.274 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: Connection refused (Connection refused)
2019-03-11 17:51:16.274 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 0 seconds (±10%)

It seems like the first thing that is logged is the connection refused warning.

Any idea what could be causing this?

Could you set the log_level to debug as described in the link I referenced earlier? My suspicion is that it did not pick up the elasticapm.properties and therefore tries to use the default url which is http://localhost8200. Have you tried to configure via environment variables?

How are you attaching the agent? Via -javaagent or via ElasticApmAttacher.attach()?

Hi Felix,

I believe the Log_level is currently set to DEBUG as seen in the elastic.properties file I posted above. Is there anything else I need to do to have debug be logged? The agent is attached using ElasticApmAttacher.attach(). It is not configured using environment variables.

Thanks for the help.

It's starting as "Main" which means you are probably starting the application from a class called Main from within your IDE. This confirms my suspicion that the elasticapm.properties is not properly picked up. Could you make a screenshot from your folder structure in your IDE? If you are using a maven-convention-based project layout, your sources should go under src/main/java, for example src/main/java/org/example/Main.java and the configuration file should be in src/main/resources/elasticapm.properties.

If you see a log line like Starting Elastic APM 1.4.0 as CXAULedgerMain

Hi Felix!

I have made the changes you advised but am still getting the timeout error, here are the newest logs:

2019-03-18 18:51:47.787 [apm-server-healthcheck] WARN co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is not available (Connection refused (Connection refused))
2019-03-18 18:51:47.806 [Attach Listener] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.4.0 as CXAULedgerMain on Java 10.0.2 (Oracle Corporation) Linux 4.18.0-1013-azure
2019-03-18 18:51:47.806 [Attach Listener] WARN co.elastic.apm.agent.configuration.StartupInfo - To enable all features and to increase startup times, please configure application_packages
Reading the settings from the file:        /cxau/settings.txt
Reading the configuration from the file:   /cxau/config.txt
Creating keys, because there are no files: /cxau/data/keys/*.pfx
This computer has an internal IP address:  172.19.0.8
2019-03-18 18:51:51 INFO All of this computer's addresses: [172.19.0.8, 127.0.0.1]
2019-03-18 18:51:51 INFO Logging initialized @4539ms to org.eclipse.jetty.util.log.Slf4jLog
2019-03-18 18:51:51 INFO Starting Javalin ...
2019-03-18 18:51:51 INFO jetty-9.4.12.v20180830; built: 2018-08-30T13:59:14.071Z; git: 27208684755d94a92186989f695db2d7b21ebc51; jvm 10.0.2+13-Debian-2
2019-03-18 18:51:51 INFO DefaultSessionIdManager workerName=node0
2019-03-18 18:51:51 INFO No SessionScavenger set, using defaults
2019-03-18 18:51:51 INFO node0 Scavenging every 660000ms
2019-03-18 18:51:51 INFO Started i.j.c.u.@45a4e269{/,null,AVAILABLE}
2019-03-18 18:51:51 INFO Started o.e.j.s.ServletContextHandler@1176e70a{/,null,AVAILABLE}
2019-03-18 18:51:51 INFO Started ServerConnector@5a451422{HTTP/1.1,[http/1.1]}{0.0.0.0:8081}
2019-03-18 18:51:51 INFO Started @4865ms
2019-03-18 18:51:51 INFO Jetty is listening on: [http://localhost:8081]
2019-03-18 18:51:51 INFO Javalin has started \o/
2019-03-18 18:52:17.863 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: Connection refused (Connection refused)
2019-03-18 18:52:17.863 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 0 seconds (±10%)
2019-03-18 18:52:47.862 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: Connection refused (Connection refused)
2019-03-18 18:52:47.862 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 1 seconds (±10%)
2019-03-18 18:53:17.861 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: Connection refused (Connection refused)
2019-03-18 18:53:17.861 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 4 seconds (±10%)
2019-03-18 18:53:47.858 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: Connection refused (Connection refused)
2019-03-18 18:53:47.858 [apm-reporter] INFO co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Backing off for 9 seconds (±10%)
2019-03-18 18:54:17.863 [apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Failed to handle event of type METRICS with this error: Connection refused (Connection refused)

Also here is the updated elasticapm.properties file:

############################################
# Core                                     #
############################################

Type: Boolean
Default value: true
active=true

Type: Boolean
Default value: true
instrument=true

# The name of your service
# This setting can not be changed at runtime. Changes require a restart of the application.
Type: String
Default value: PlugableMain
service_name=PlugableMain

# A version string for the currently deployed version of the service. If you don’t version your deployments, the recommended value for this field is the commit identifier of the deployed revision, e.g. the output of git rev-parse HEAD.
#
# This setting can not be changed at runtime. Changes require a restart of the application.
# Type: String
# Default value:
#
service_version= 1.0

# The name of the environment this service is deployed in, e.g. "production" or "staging".
Type: String
#Default value:
environment=test

# By default, the agent will sample every transaction (e.g. request to your service). To reduce overhead and storage requirements, you can set the sample rate to a value between 0.0 and 1.0. We still record overall time and the result for unsampled transactions, but no context information, tags, or spans.
Type: Double
#Default value: 1.0
transaction_sample_rate=1.0

# Limits the amount of spans that are recorded per transaction.
# This is helpful in cases where a transaction creates a very high amount of spans (e.g. thousands of SQL queries).
# Setting an upper limit will prevent overloading the agent and the APM server with too much work for such edge cases.
# This setting can be changed at runtime
Type: Integer
#Default value: 500
transaction_max_spans=500


Type: Level
#Default value: INFO
log_level=DEBUG

#This setting can not be changed at runtime. Changes require a restart of the application.
Type: comma separated list
Default value: https://7aea994c47fc41749d1faa5a9e48b604.apm.us-east-1.aws.cloud.es.io:443
server_urls= https://7aea994c47fc41749d1faa5a9e48b604.apm.us-east-1.aws.cloud.es.io:443
# This setting can not be changed at runtime. Changes require a restart of the application.
# Type: String
# Default value:
secret_token=XXXXXXXXXXXXXXXXX

Lastly here is a screenshot of the new folder structure:
image

Do you think it is that I am missing something in the properties file? Is there a way to determine if the properties file is being read? Let me know what ideas you have.

Thanks much for all the help!

What's a bit strange about your properties file is that some things are not commented out which are supposed to be commented out:

Could you start your application with -Delastic.apm.log_level=debug and paste your logs here? It seems it still can't find the properties file. Are you starting from within the IDE or via java -jar? If you are starting via the IDE, maybe it is not properly configured to pick up the resources? One way to verify this is to add this line to your main method:

System.out.println(MyMainClass.class.getResource("/elasticapm.properties").getPath());

If it throws a NullPointerException, it is not properly configured. Otherwise it should print the path to the file.

If you are starting via java -jar, can you verify that the jar actually contains the properties file?

Note that a properties file within your application only works when you attach via ElasticApmAttacher.attach(). It does not work with -javaagent attachment. In this case, you would have to place the properties file in the same folder as the agent jar.

Hi Felix!

I have made the changes you advised to the properties file so that those lines are commented out.

It is setup using java -jar and and that the file is in the same folder as you described. I set the logs to debug and will include them here as a link to a file given how big they are, although I did not see "NullPointerException"

Let me know what you think of this, and I can try any suggestions. Thanks for all your help.

Link: https://drive.google.com/open?id=1YpP0WcsBDaBrORbX1QJ4UNLcGYegTAeB

The logs seem to suggest it’s working now?

Hi Felix!

Yes they are indeed now coming in thanks for the help with that. I see now the apm* index, but when going under APM I do not see any services or traces. Performing a query for any of the logs that I see in the "discover" section does not bring any results back.

Any ideas why no services are appearing? I made sure to add in the objects and it found 16.

Much appreciated!

Not sure if that's what you meant by "I made sure to add in the objects and it found 16." but did you Load Kibana objects? See https://www.elastic.co/guide/en/kibana/6.6/apm-getting-started.html?

If it still does not work, could you execute this query in the Kibana Dev Tools console:

GET /apm*/_search
{
  "query": {
    "match_all": {}
  }
}

Sorry for not being clear, yes I had loaded kibana objects, 16 objects were found.

When executing the query you mentioned I get this response back:

Here is a link to the code (too large to paste here):

Let me know what you think.

Cheers!

Could you send me the result of this query?

GET /apm*/_search
{
  "query": {
    "match_all": {}
  },
  "size": 0
  , "aggs": {
    "type": {
      "terms": {
        "field": "processor.event",
        "size": 10
      }
    }
  }
}

Sure thing, here it is:

{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 4,
    "successful" : 4,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 268,
    "max_score" : 0.0,
    "hits" : [ ]
  },
  "aggregations" : {
    "type" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "metric",
          "doc_count" : 267
        },
        {
          "key" : "onboarding",
          "doc_count" : 1
        }
      ]
    }
  }
}

Hi @felixbarny ! Hope you had a good weekend.

Any updates on what may be causing no services or traces to appear?

Thanks!

Hi and thanks for the ping! It seems like the connection works now but no tracing data is coming in. The most likely reason is that you are not using one of the supported web frameworks. Could you tell me which web framework you are using?

Hi @felixbarny ,

We are using the javalin.io framework. Would this be a problem?

Thanks!

Hi Javier,

we don't have out-of-the-box support for Javalin, unfortunately. But you can use the public API to manually create transactions.

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.