APM Server error (Connection refused)

[Krish007 (Krishna)]

APM Server version: 7.3.2

APM Agent language and version: Elastic.Apm.NetCoreAll (1.0.1)

Original install method (e.g. download page, yum, deb, from source, etc.) and version: docker (18.09.9)

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant): APM server error : Always getting error message saying connection refused.

I am always getting error message saying
ERROR pipeline/output.go:100
Failed to connect to backoff(elasticsearch(http://elasticsearch:9200)): 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/]","header":{"WWW-Authenticate":"Basic realm="security" charset="UTF-8""}},"status":401}

I am trying to setup elasticsearch, kibanan and APM server using docker. I am able to run elasticsearch and kibana successfully. But not able to run APM server.

My docker-compose file is

version: '3'

services:

elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.3.2
    container_name: elasticsearch
    restart: always
    environment:
        - cluster.name=docker-cluster
        - bootstrap.memory_lock=true
        - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
        - xpack.security.enabled=true
        - discovery.type=single-node
        - ELASTIC_PASSWORD=PleaseChangeMe
    ulimits:
        memlock:
            soft: -1
            hard: -1
    ports:
        - "9200:9200"
    expose:
        - "9200"
kibana:
    image: docker.elastic.co/kibana/kibana:7.3.2
    container_name: kibana
    hostname: kibana
    restart: always
    environment:
        - SERVER_NAME=kibana.localhost
        - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
        - XPACK_SECURITY_ENABLED=true
        - ELASTICSEARCH_USERNAME=elastic
        - ELASTICSEARCH_PASSWORD=PleaseChangeMe
    ports:
        - "5601:5601"
    expose:
        - "5601"
    links:
        - elasticsearch
    depends_on:
        - elasticsearch
apm:
    image: docker.elastic.co/apm/apm-server:7.3.2
    environment:
        - output.elasticsearch.enabled=true
        - output.elasticsearch.hosts=http://elasticsearch:9200
        - output.elasticsearch.username=elastic
        - output.elasticsearch.password=PleaseChangeMe
        - cloud.auth=elastic:PleaseChangeMe
    ports:
        - "8200:8200"
    expose:
        - "8200"
    links:
        - elasticsearch
    depends_on:
        - elasticsearch

Looks like I am passing elasticsearch credentials in the wrong way.

[jalvz (Juan Alvarez)]

Hi,

the environment block in the apm section is not working as you expect. Instead, you need something like this:

   image: docker.elastic.co/apm/apm-server:7.3.2
   command: [
        "apm-server", 
        "-e", 
        "-E", "output.elasticsearch.username=elastic", 
        "-E", "output.elasticsearch.password=PleaseChangeMe", 
    ]
    ....

Also note that cloud.auth overrides output.elasticsearch.username and output.elasticsearch.password, so use one or the other, but not both.

Hope that helps!

[Krish007 (Krishna)]

Hi jalvz,

Thanks for your reply..
Now it is working fine

But when I am running the dot net core application after integrating apm agent(Elastic.Apm.NetCoreAll), I am getting warning message saying.

{{Scope}} Failed capturing request (failed to remove from ProcessingRequests) - This Span will be skipped in case it wasn't captured before. Request: method: "POST", URL: http://elastic:PleaseChangeMe@localhost:9200/_bulk

I am not sure why I am getting this error message, Did I miss any configuration in elasticsearch ?

[jalvz (Juan Alvarez)]

Hi,

You are seeing that log In your .NET app? That URL is Elasticsearch, you have to point the agent to APM Server, is that the issue?

[Sergey_Kleyman]

Hi @Krish007 - welcome to our community!

Does the application (which you monitor with Elastic APM .NET Agent) use Elasticsearch client?
It seems that the HTTP localhost:9200/_bulk request comes from Elasticsearch client for .NET.

[Krish007 (Krishna)]

Yes Jalvz..This log is from .NET App.
Yes...I have pointed the APM agent to APM server and It's working fine for me. I mean I am able to visualize APM in Kibana also able to see all the application transactions.

Even though I am able to see application transactions in Kibana, I am also getting above said warning message in .NET App continuously..I hope elasticsearch is not supporting bulk posting API. Did I miss anything ?

[Krish007 (Krishna)]

Yes Sergey..My .NET App is also using elasticsearch to save application logs (Warnings/Errors/ Custom Logs/ Event Logs etc.).

Thing is that I am getting this warning message only after the integration of APM agent in .NET App. Looks like APM agent in .NET App is trying to make call to the ( HTTP localhost:9200/_bulk) elasticserver.

[Sergey_Kleyman]

HTTP call localhost:9200/_bulk itself is actually coming from Elasticsearch .NET Client that you use in your .NET app. Elasticsearch .NET Client uses HTTP protocol to communicate with Elasticsearch cluster. But the Warning log message Failed capturing request (failed to remove from ProcessingRequests) - This Span will be skipped in case it wasn't captured before is not from Elasticsearch .NET client - this message is printed by Elastic APM .NET Agent that tries to record some info (start time, duration, etc.) about all HTTP calls that application makes which includes HTTP calls that libraries that application uses make and in your case one of those libraries is Elasticsearch .NET Client.

The meaning of this message is that Agent sees the end of some HTTP request but it cannot find the corresponding start event (to calculate duration, etc.) We still work on thorough support for tracing of Elasticsearch .NET Client in Elastic APM .NET Agent (https://github.com/elastic/apm-agent-dotnet/pull/329) but even without that full support Agent should not encounter such issues (not being able to find the corresponding start event) - I've opened a bug issue (https://github.com/elastic/apm-agent-dotnet/issues/500) to investigate. In the meantime if you would like to silence those log messages you can change Elastic APM .NET Agent log level to Error as described in the documentation.

[Krish007 (Krishna)]

Thanks Sergey for your reply.