APM Server - Error Connection Refused

rtpatel95 · April 16, 2019, 4:01pm

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text

Kibana version:
6.7.1

Elasticsearch version:
6.7.1

APM Server version:
6.7.1

APM Agent language and version:
English
elastic-apm-rum.umd.min.js v4.0.1 from github

Browser version:
Chrome 71.0.3578.98

OS Version:
Ubuntu 18.04 LTS

Original install method (e.g. download page, yum, deb, from source, etc.) and version:
APT Repository

Fresh install or upgraded from other version?
Fresh

Is there anything special in your setup?

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
Trying to setup up APM on simple Flask App. Python agent is working fine and we can see transactions in APM. The client-side is not interacting at all with the APM-Server. We are receiving ERR_CONNECTION_REFUSED error in the Chrome Debugger. Even though we have enabled RUM in the config file.

Steps to reproduce:
1.Setup basic Flask App with only a blank button (button generates a random number sends it to python, in python number is squared then displayed in an alert on web app).
2.Click button.
3.Receive alert of randomly generated integer squared.
4.Look in Chrome DevTools Console to see error.

Errors in browser console (if relevant):
elastic-apm-rum.umd.min.js:2 OPTIONS http://localhost:8200/intake/v2/rum/events net::ERR_CONNECTION_REFUSED

(anonymous)	@	elastic-apm-rum.umd.min.js:2
(anonymous)	@	elastic-apm-rum.umd.min.js:2
r.(anonymous function)	@	elastic-apm-rum.umd.min.js:2
(anonymous)	@	elastic-apm-rum.umd.min.js:2
e._makeHttpRequest	@	elastic-apm-rum.umd.min.js:2
e._postJson	@	elastic-apm-rum.umd.min.js:2
e.sendTransactions	@	elastic-apm-rum.umd.min.js:2
(anonymous)	@	elastic-apm-rum.umd.min.js:2
e.flush	@	elastic-apm-rum.umd.min.js:2
(anonymous)	@	elastic-apm-rum.umd.min.js:2
setTimeout (async)
e._setTimer	@	elastic-apm-rum.umd.min.js:2
e.add	@	elastic-apm-rum.umd.min.js:2
(anonymous)	@	elastic-apm-rum.umd.min.js:2
e.addTransaction	@	elastic-apm-rum.umd.min.js:2
(anonymous)	@	elastic-apm-rum.umd.min.js:2
(anonymous)	@	elastic-apm-rum.umd.min.js:2
e.applyAll	@	elastic-apm-rum.umd.min.js:2
e.add	@	elastic-apm-rum.umd.min.js:2
(anonymous)	@	elastic-apm-rum.umd.min.js:2
(anonymous)	@	elastic-apm-rum.umd.min.js:2
Promise.then (async)
(anonymous)	@	elastic-apm-rum.umd.min.js:2
e.runOuter	@	elastic-apm-rum.umd.min.js:2
e.applyAsync	@	elastic-apm-rum.umd.min.js:2
a.onEnd	@	elastic-apm-rum.umd.min.js:2
e.callOnEnd	@	elastic-apm-rum.umd.min.js:2
r.end	@	elastic-apm-rum.umd.min.js:2
r.detectFinish	@	elastic-apm-rum.umd.min.js:2
r.removeTask	@	elastic-apm-rum.umd.min.js:2
(anonymous)	@	elastic-apm-rum.umd.min.js:2
setTimeout (async)
i	@	elastic-apm-rum.umd.min.js:2
load (async)
e._sendPageLoadMetrics	@	elastic-apm-rum.umd.min.js:2
e.init	@	elastic-apm-rum.umd.min.js:2
(anonymous)	@	(index):9

elastic-apm-rum.umd.min.js:2 Failed sending transactions! Error: http://localhost:8200/intake/v2/rum/events HTTP status: 0 at XMLHttpRequest.i.onreadystatechange (elastic-apm-rum.umd.min.js:2)

o.(anonymous function)	@	elastic-apm-rum.umd.min.js:2
(anonymous)	@	elastic-apm-rum.umd.min.js:2
Promise.then (async)
(anonymous)	@	elastic-apm-rum.umd.min.js:2
e.flush	@	elastic-apm-rum.umd.min.js:2
(anonymous)	@	elastic-apm-rum.umd.min.js:2
setTimeout (async)
e._setTimer	@	elastic-apm-rum.umd.min.js:2
e.add	@	elastic-apm-rum.umd.min.js:2
(anonymous)	@	elastic-apm-rum.umd.min.js:2
e.addTransaction	@	elastic-apm-rum.umd.min.js:2
(anonymous)	@	elastic-apm-rum.umd.min.js:2
(anonymous)	@	elastic-apm-rum.umd.min.js:2
e.applyAll	@	elastic-apm-rum.umd.min.js:2
e.add	@	elastic-apm-rum.umd.min.js:2
(anonymous)	@	elastic-apm-rum.umd.min.js:2
(anonymous)	@	elastic-apm-rum.umd.min.js:2
Promise.then (async)
(anonymous)	@	elastic-apm-rum.umd.min.js:2
e.runOuter	@	elastic-apm-rum.umd.min.js:2
e.applyAsync	@	elastic-apm-rum.umd.min.js:2
a.onEnd	@	elastic-apm-rum.umd.min.js:2
e.callOnEnd	@	elastic-apm-rum.umd.min.js:2
r.end	@	elastic-apm-rum.umd.min.js:2
r.detectFinish	@	elastic-apm-rum.umd.min.js:2
r.removeTask	@	elastic-apm-rum.umd.min.js:2
(anonymous)	@	elastic-apm-rum.umd.min.js:2
setTimeout (async)
i	@	elastic-apm-rum.umd.min.js:2
load (async)
e._sendPageLoadMetrics	@	elastic-apm-rum.umd.min.js:2
e.init	@	elastic-apm-rum.umd.min.js:2
(anonymous)	@	(index):9

Provide logs and/or server output (if relevant):

Hamidreza · April 24, 2019, 11:59am

Hi,

Welcome to our forum.

It seems that the preflight (OPTIONS) request is failing. This could be because of misconfiguration on the APM Server.
Would you please post the content of your APM Server configuration file?

Cheers,
Hamid

rtpatel95 · April 24, 2019, 8:04pm

################### APM Server Configuration #########################

############################# APM Server ######################################

apm-server:

Defines the host and port the server is listening on. use "unix:/path/to.sock" to listen on a unix domain socket.

host: "localhost:8200"

Maximum permitted size in bytes of a request's header accepted by the server to be processed.

#max_header_size: 1048576

Maximum permitted duration for reading an entire request.

#read_timeout: 30s

Maximum permitted duration for writing a response.

#write_timeout: 30s

Maximum duration in seconds before releasing resources when shutting down the server.

#shutdown_timeout: 5s

#-- v1 Intake API (deprecated)

Maximum permitted size in bytes of an unzipped request accepted by the server to be processed.

#max_unzipped_size: 31457280

Maximum duration request will be queued before being read.

#max_request_queue_time: 2s

Maximum number of requests permitted to be sent to the server concurrently.

#concurrent_requests: 5

#-- v2 Intake API

Maximum allowed size in bytes of a single event

#max_event_size: 307200

#--

Maximum number of new connections to accept simultaneously (0 means unlimited)

max_connections: 0

Authorization token to be checked. If a token is set here the agents must

send their token in the following format: Authorization: Bearer .

It is recommended to use an authorization token in combination with SSL enabled,

and save the token in the beats keystore.

secret_token: "HCBE"
#ssl.enabled: false
#ssl.certificate : "path/to/cert"
#ssl.key : "path/to/private_key"

It is recommended to use the provided keystore instead of entering the passphrase in plain text.

#ssl.key_passphrase: ""

rtpatel95 · April 24, 2019, 8:04pm

rum:
# To enable real user monitoring (RUM) support set this to true.
enabled: true

#-- v1 RUM endpoint (deprecated)

# Rate limit per second and IP address for requests sent to the RUM endpoint.
#rate_limit: 10

#-- v2 RUM endpoint

#event_rate:

  # Defines the maximum amount of events allowed to be sent to the APM Server v2 RUM
  # endpoint per ip per second. Defaults to 300.
  #limit: 300

  # An LRU cache is used to keep a rate limit per IP for the most recently seen IPs.
  # This setting defines the number of unique IPs that can be tracked in the cache.
  # Sites with many concurrent clients should consider increasing this limit. Defaults to 1000.
  #lru_size: 1000

#-- General RUM settings

# Comma separated list of permitted origins for real user monitoring.
# User-agents will send an origin header that will be validated against this list.
# An origin is made of a protocol scheme, host and port, without the url path.
# Allowed origins in this setting can have * to match anything (eg.: http://*.example.com)
# If an item in the list is a single '*', everything will be allowed
allow_origins : ['*']

# Regexp to be matched against a stacktrace frame's `file_name` and `abs_path` attributes.
# If the regexp matches, the stacktrace frame is considered to be a library frame.
#library_pattern: "node_modules|bower_components|~"

# Regexp to be matched against a stacktrace frame's `file_name`.
# If the regexp matches, the stacktrace frame is not used for calculating error groups.
# The default pattern excludes stacktrace frames that have a filename starting with '/webpack'
#exclude_from_grouping: "^/webpack"

# If a source map has previously been uploaded, source mapping is automatically applied
# to all error and transaction documents sent to the RUM endpoint.
#source_mapping:

  # Source maps are always fetched from Elasticsearch, by default using the output.elasticsearch configuration.
  # A different instance must be configured when using any other output.
  # This setting only affects sourcemap reads - the output determines where sourcemaps are written.
  #elasticsearch:
    # Array of hosts to connect to.
    # Scheme and port can be left out and will be set to the default (http and 9200)
    # In case you specify and additional path, the scheme is required: http://localhost:9200/path
    # IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
    # hosts: ["localhost:9200"]

    # Optional protocol and basic auth credentials.
    #protocol: "https"
    #username: "elastic"
    #password: "changeme"

  # The `cache.expiration` determines how long a source map should be cached before fetching it again from Elasticsearch.
  # Note that values configured without a time unit will be interpreted as seconds.
  #cache:
    #expiration: 5m

  # Source maps are stored in a seperate index.
  # If the default index pattern for source maps at 'outputs.elasticsearch.indices'
  # is changed, a matching index pattern needs to be specified here.
  #index_pattern: "apm-*-sourcemap*"

If set to true, APM Server augments data received by the agent with the original IP of the backend server,

or the IP and User Agent of the real user (RUM requests). It defaults to true.

#capture_personal_data: true

golang expvar support - https://golang.org/pkg/expvar/

#expvar:
# Set to true to Expose expvar
#enabled: false

# Url to expose expvar
#url: "/debug/vars"

Instrumentation support for the server's HTTP endpoints and event publisher.

#instrumentation:
# Set to true to enable instrumentation of the APM server itself.
#enabled: false
# Environment in which the APM Server is running on (eg: staging, production, etc.)
#environment: ""
# Remote host to report instrumentation results to. Single entry permitted until
# https://github.com/elastic/apm-agent-go/issues/200 is resolved.
#hosts:
# - http://remote-apm-server:8200
# Remote apm-servers' secret_token
#secret_token:

Metrics endpoint

#metrics:
# Set to false to disable the metrics endpoint
#enabled: true

A pipeline is a definition of processors applied to documents when writing them to Elasticsearch.

Using pipelines involves two steps:

(1) registering a pipeline

(2) applying a pipeline during data ingestion (see `output.elasticsearch.pipelines`)

You can manually register pipelines, or use this configuration option to ensure

pipelines are loaded and registered at the configured Elasticsearch instances.

Automatic pipeline registration requires

* `output.elasticsearch` to be enabled and configured.

* having the required Elasticsearch Processor Plugins installed.

APM Server default pipelines require you to have the `Ingest User Agent Plugin` installed.

Find the default pipeline configuration at `ingest/pipeline/definition.json`.

#register.ingest.pipeline:
# Registers pipeline definitions in Elasticsearch on APM Server startup. Defaults to false.
#enabled: false

# Overwrites existing pipeline definitions in Elasticsearch. Defaults to true.
#overwrite: true

rtpatel95 · April 24, 2019, 8:05pm

#================================ General ======================================

Internal queue configuration for buffering events to be published.

#queue:

Queue type by name (default 'mem')

The memory queue will present all available events (up to the outputs

bulk_max_size) to the output, the moment the output is ready to server

another batch of events.

#mem:
# Max number of events the queue can buffer.
#events: 4096

# Hints the minimum number of events stored in the queue,
# before providing a batch of events to the outputs.
# A value of 0 (the default) ensures events are immediately available
# to be sent to the outputs.
#flush.min_events: 2048

# Maximum duration after which events are available to the outputs,
# if the number of events stored in the queue is < min_flush_events.
#flush.timeout: 1s

Sets the maximum number of CPUs that can be executing simultaneously. The

default is the number of logical CPUs available in the system.

#max_procs:

#============================== Template =====================================

A template is used to set the mapping in Elasticsearch

By default template loading is enabled and the template is loaded.

These settings can be adjusted to load your own template or overwrite existing ones.

Set to false to disable template loading.

#setup.template.enabled: true

Template name. By default the template name is "apm-%{[beat.version]}"

The template name and pattern has to be set in case the elasticsearch index pattern is modified.

#setup.template.name: "apm-%{[beat.version]}"

Template pattern. By default the template pattern is "apm-%{[beat.version]}-*" to apply to the default index settings.

The first part is the version of the beat and then -* is used to match all daily indices.

The template name and pattern has to be set in case the elasticsearch index pattern is modified.

#setup.template.pattern: "apm-%{[beat.version]}-*"

Path to fields.yml file to generate the template

#setup.template.fields: "${path.config}/fields.yml"

Overwrite existing template

#setup.template.overwrite: false

Elasticsearch template settings

#setup.template.settings:

A dictionary of settings to place into the settings.index dictionary

of the Elasticsearch template. For more details, please check

https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html

#index:
#number_of_shards: 1
#codec: best_compression
#number_of_routing_shards: 30
#mapping.total_fields.limit: 2000

A dictionary of settings for the _source field. For more details, please check

https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html

#_source:
#enabled: false

rtpatel95 · April 24, 2019, 8:05pm

#============================== Deprecated: Dashboards =====================================

Deprecated: Loading dashboards from the APM Server into Kibana is deprecated from 6.4 on.

We suggest to use the Kibana UI to load APM Server dashboards and index pattern instead.

These settings control loading the sample dashboards to the Kibana index. Loading

the dashboards are disabled by default and can be enabled either by setting the

options here, or by using the `-setup` CLI flag or the `setup` command.

#setup.dashboards.enabled: false

The directory from where to read the dashboards. The default is the `kibana`

folder in the home path.

#setup.dashboards.directory: ${path.home}/kibana

The URL from where to download the dashboards archive. It is used instead of

the directory if it has a value.

#setup.dashboards.url:

The file archive (zip file) from where to read the dashboards. It is used instead

of the directory when it has a value.

#setup.dashboards.file:

The name of the Kibana index to use for setting the configuration. Default is ".kibana"

#setup.dashboards.kibana_index: .kibana

The Elasticsearch index name. This overwrites the index name defined in the

dashboards and index pattern. Example: testbeat-*

The dashboards.index needs to be changed in case the elasticsearch index pattern is modified.

#setup.dashboards.index:

Always use the Kibana API for loading the dashboards instead of autodetecting

how to install the dashboards by first querying Elasticsearch.

#setup.dashboards.always_kibana: false

If true and Kibana is not reachable at the time when dashboards are loaded,

it will retry to reconnect to Kibana instead of exiting with an error.

#setup.dashboards.retry.enabled: false

Duration interval between Kibana connection retries.

#setup.dashboards.retry.interval: 1s

Maximum number of retries before exiting with an error, 0 for unlimited retrying.

#setup.dashboards.retry.maximum: 0

#============================== Deprecated: Kibana =====================================

Deprecated: Starting with APM Server version 6.4, loading dashboards and index pattern

from the APM Server into Kibana is deprecated.

We suggest to use the Kibana UI to load APM Server dashboards and index pattern instead.

Setting up a Kibana endpoint is not necessary when loading the index pattern and dashboards via the UI.

#setup.kibana:

Kibana Host

Scheme and port can be left out and will be set to the default (http and 5601)

In case you specify and additional path, the scheme is required: http://localhost:5601/path

IPv6 addresses should always be defined as: https://[2001:db8::1]:5601

#host: "localhost:5601"

Optional protocol and basic auth credentials.

#protocol: "https"
#username: "elastic"
#password: "changeme"

Optional HTTP Path

#path: ""

Use SSL settings for HTTPS. Default is false.

#ssl.enabled: false

Configure SSL verification mode. If `none` is configured, all server hosts

and certificates will be accepted. In this mode, SSL based connections are

susceptible to man-in-the-middle attacks. Use only for testing. Default is

`full`.

#ssl.verification_mode: full

List of supported/valid TLS versions. By default all TLS versions 1.0 up to

1.2 are enabled.

#ssl.supported_protocols: [TLSv1.0, TLSv1.1, TLSv1.2]

SSL configuration. By default is off.

List of root certificates for HTTPS server verifications

#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

Certificate for SSL client authentication

#ssl.certificate: "/etc/pki/client/cert.pem"

Client Certificate Key

#ssl.key: "/etc/pki/client/cert.key"

Optional passphrase for decrypting the Certificate Key.

It is recommended to use the provided keystore instead of entering the passphrase in plain text.

#ssl.key_passphrase: ''

Configure cipher suites to be used for SSL connections

#ssl.cipher_suites:

Configure curve types for ECDHE based cipher suites

#ssl.curve_types:

rtpatel95 · April 24, 2019, 8:06pm

IF you want a txt file so its easier to read, please provide me with email and I will send it too you

Hamidreza · April 25, 2019, 7:54am

Hi,

Are you running your browser in a different machine to the one running APM Server? If that's the case you should set host:0.0.0.0:8200 to make sure APM Server is accessible remotely. Furthermore you should configure the RUM agent with the correct serverUrl.

I've also send you my email on the PM.

Cheers,
Hamid

rtpatel95 · April 25, 2019, 2:42pm

Hi Hamid,

Still having issues, sent email to further clarify and attach config file.

system · May 16, 2019, 10:42am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.

APM Server - Error Connection Refused

Defines the host and port the server is listening on. use "unix:/path/to.sock" to listen on a unix domain socket.

Maximum permitted size in bytes of a request's header accepted by the server to be processed.

Maximum permitted duration for reading an entire request.

Maximum permitted duration for writing a response.

Maximum duration in seconds before releasing resources when shutting down the server.

Maximum permitted size in bytes of an unzipped request accepted by the server to be processed.

Maximum duration request will be queued before being read.

Maximum number of requests permitted to be sent to the server concurrently.

Maximum allowed size in bytes of a single event

Maximum number of new connections to accept simultaneously (0 means unlimited)

max_connections: 0

Authorization token to be checked. If a token is set here the agents must

send their token in the following format: Authorization: Bearer .

It is recommended to use an authorization token in combination with SSL enabled,

and save the token in the beats keystore.

It is recommended to use the provided keystore instead of entering the passphrase in plain text.

If set to true, APM Server augments data received by the agent with the original IP of the backend server,

or the IP and User Agent of the real user (RUM requests). It defaults to true.

golang expvar support - https://golang.org/pkg/expvar/

Instrumentation support for the server's HTTP endpoints and event publisher.

Metrics endpoint

A pipeline is a definition of processors applied to documents when writing them to Elasticsearch.

Using pipelines involves two steps:

(1) registering a pipeline

(2) applying a pipeline during data ingestion (see output.elasticsearch.pipelines)

You can manually register pipelines, or use this configuration option to ensure

pipelines are loaded and registered at the configured Elasticsearch instances.

Automatic pipeline registration requires

* output.elasticsearch to be enabled and configured.

* having the required Elasticsearch Processor Plugins installed.

APM Server default pipelines require you to have the Ingest User Agent Plugin installed.

Find the default pipeline configuration at ingest/pipeline/definition.json.

Internal queue configuration for buffering events to be published.

Queue type by name (default 'mem')

The memory queue will present all available events (up to the outputs

bulk_max_size) to the output, the moment the output is ready to server

another batch of events.

Sets the maximum number of CPUs that can be executing simultaneously. The

default is the number of logical CPUs available in the system.

A template is used to set the mapping in Elasticsearch

By default template loading is enabled and the template is loaded.

These settings can be adjusted to load your own template or overwrite existing ones.

Set to false to disable template loading.

Template name. By default the template name is "apm-%{[beat.version]}"

The template name and pattern has to be set in case the elasticsearch index pattern is modified.

Template pattern. By default the template pattern is "apm-%{[beat.version]}-*" to apply to the default index settings.

The first part is the version of the beat and then -* is used to match all daily indices.

The template name and pattern has to be set in case the elasticsearch index pattern is modified.

Path to fields.yml file to generate the template

Overwrite existing template

Elasticsearch template settings

A dictionary of settings to place into the settings.index dictionary

of the Elasticsearch template. For more details, please check

https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html

A dictionary of settings for the _source field. For more details, please check

https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-source-field.html

Deprecated: Loading dashboards from the APM Server into Kibana is deprecated from 6.4 on.

We suggest to use the Kibana UI to load APM Server dashboards and index pattern instead.

These settings control loading the sample dashboards to the Kibana index. Loading

the dashboards are disabled by default and can be enabled either by setting the

options here, or by using the -setup CLI flag or the setup command.

The directory from where to read the dashboards. The default is the kibana

folder in the home path.

The URL from where to download the dashboards archive. It is used instead of

the directory if it has a value.

The file archive (zip file) from where to read the dashboards. It is used instead

of the directory when it has a value.

The name of the Kibana index to use for setting the configuration. Default is ".kibana"

The Elasticsearch index name. This overwrites the index name defined in the

dashboards and index pattern. Example: testbeat-*

The dashboards.index needs to be changed in case the elasticsearch index pattern is modified.

Always use the Kibana API for loading the dashboards instead of autodetecting

how to install the dashboards by first querying Elasticsearch.

If true and Kibana is not reachable at the time when dashboards are loaded,

it will retry to reconnect to Kibana instead of exiting with an error.

Duration interval between Kibana connection retries.

Maximum number of retries before exiting with an error, 0 for unlimited retrying.

Deprecated: Starting with APM Server version 6.4, loading dashboards and index pattern

from the APM Server into Kibana is deprecated.

(2) applying a pipeline during data ingestion (see `output.elasticsearch.pipelines`)

* `output.elasticsearch` to be enabled and configured.

APM Server default pipelines require you to have the `Ingest User Agent Plugin` installed.

Find the default pipeline configuration at `ingest/pipeline/definition.json`.

options here, or by using the `-setup` CLI flag or the `setup` command.

The directory from where to read the dashboards. The default is the `kibana`

Configure SSL verification mode. If `none` is configured, all server hosts

`full`.