No access to APM API

dthapostle · April 13, 2022, 8:51am

IKibana version: 7.17.2

Elasticsearch version: 7.17.2

APM Server version: APM integration 7.17.1

APM Agent language and version: ASP.Net 7.17.2

Browser version: Safari 15.3

Original install method (e.g. download page, yum, deb, from source, etc.) and version: deb/apt

Fresh install or upgraded from other version? fresh installation

Is there anything special in your setup? ElasticAgent is configured to interact with ES and fleet directly.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
I configured APM integration following the manual located in APM | Kibana Guide [7.17] | Elastic

I can't access APM API:

Steps to reproduce:

Install ELK v7.17
Install Elastic APM integration following elastic.co manual
Try to access APM API - get 404

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):
22:09:05.916

[elastic_agent][error] Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post "https://elastic3.***:8220/api/fleet/agents/4f590859-53da-4d56-a605-c6bff181389d/checkin?": context canceled

22:09:05.916

[elastic_agent][error] Could not communicate with fleet-server Checking API will retry, error: fail to checkin to fleet-server: Post "https://elastic3.***:8220/api/fleet/agents/4f590859-53da-4d56-a605-c6bff181389d/checkin?": context canceled

simitt · April 20, 2022, 9:41am

Hi @dthapostle ,
the APM Agent Configuration API is available via the Kibana endpoint, not the Fleet Server endpoint. Can you please try with that and let us know if it works.

Thanks,
Silvia

dthapostle · April 20, 2022, 10:28am

@simitt

Thanks for reply.
Using Kibana endpoint I can proceed with agents configurations:
http://logcollect.****.com:5601/api/apm/settings/agent-configuration

{
    "configurations": [
        {
            "service": {},
            "settings": {},
            "@timestamp": 1649784140647,
            "applied_by_agent": true,
            "etag": "05c712546752f5b1fe3b429f84df5d38827a2fdf"
        }
    ]
}

But ASP Test app can't establish connection to config:

simitt · April 20, 2022, 1:47pm

The apm agent configuration settings can be set and updated directly via Kibana, but are consumed by the APM agents through the APM Server. This way the APM agents do not require a direct connection to Kibana.

So for consuming the APM agent configuration, you need to configure the APM Server URL (not the Kibana URL) in the agents. The APM Server URL defaults to port 8200.

dthapostle · April 20, 2022, 2:07pm

@simitt thanks.
How should I configure it for internal APM server? It seems to ignore apm-server.yml hosted in the same folder:

root@elastic3:/var/lib/elasticsearch# ps ax | grep ap[m]
 212053 ?        Sl    10:16 /var/lib/elastic-agent/data/elastic-agent-6118f2/install/apm-server-8.1.2-linux-x86_64/apm-server -E management.enabled=true -E gc_percent=${APMSERVER_GOGC:100} -E logging.level=info -E http.enabled=true -E http.host=unix:///var/lib/elastic-agent/data/tmp/default/apm-server/apm-server.sock -E logging.files.path=/var/lib/elastic-agent/data/elastic-agent-6118f2/logs/default -E logging.files.name=apm-server -E logging.files.keepfiles=7 -E logging.files.permission=0640 -E logging.files.interval=1h -E path.data=/var/lib/elastic-agent/data/elastic-agent-6118f2/run/default/apm-server--8.1.2

configuration file:

root@elastic3:/var/lib/elastic-agent/data/elastic-agent-6118f2/install/apm-server-8.1.2-linux-x86_64# grep host apm-server.yml  | head -n 5
  # Defines the host and port the server is listening on. Use "unix:/path/to.sock" to listen on a unix domain socket.
  host: "elastic3.****:8200"
    # An origin is made of a protocol scheme, host and port, without the url path.
        # Array of hosts to connect to.
        # In case you specify and additional path, the scheme is required: `http://localhost:9200/path`.

simitt · April 20, 2022, 3:02pm

The APM agents should not require any specific configuration, the central config management is enabled by default and the apm agent is fetching the information from the configured APM Server URL.
The shared screenshot in No access to APM API - #2 by simitt suggested that you had the Kibana URL configured in your .NET apm agent. When setting that to the APM Server URL (see this config example), the .NET agent should be good to fetch the information from the APM Server.

You shouldn't have to configure anything in the Elastic Agent related to this. When running APM Server managed by Elastic Agent, the APM central configuration settings are automatically passed down from Kibana to the APM Server via the Fleet/Elastic Agent mechanism.

So in summary, please ensure that you

use the Kibana path for setting the concrete apm agent central config values
configure the APM Server URL in the .NET apm agent

No further config should be required related to this Feature.

dthapostle · April 20, 2022, 3:32pm

@simitt
I have 3 ES nodes: logcollect, elastic2 and elastic3. Elastic agent is installed on elastic3 with ElasticAPM (via Agent policy). There's APM process on elastic3 but it doesn't listen TCP 8200.
I can't specify elastic3:8200 in .Net APM agent because built-in APM server on elastic3 doesn't listen TCP 8200:

root@elastic3:~# netstat -ano | grep 82[0,2]0
tcp        0      0 192.168.152.24:45272    192.168.152.24:8220     ESTABLISHED keepalive (8.10/0/0)
tcp6       0      0 :::8220                 :::*                    LISTEN      off (0.00/0/0)
tcp6       0      0 192.168.152.24:8220     192.168.152.24:45272    ESTABLISHED keepalive (8.10/0/0)
tcp6       0      0 192.168.152.24:8220     192.168.152.76:59210    ESTABLISHED keepalive (1.79/0/0)
root@elastic3:~#

simitt · April 20, 2022, 4:28pm

Please follow the guide to add the APM Integration. When running inside a container, also ensure to configure the host via the Fleet UI to listen to all interfaces, see this example:

dthapostle · April 20, 2022, 4:50pm

@simitt I specified for fleet and client policies:

and restarted the agent - nobody listens TCP 8200:

root@elastic3:~# service elastic-agent restart
root@elastic3:~# netstat -anpo | grep 82[0,2]0
tcp        0      0 192.168.152.24:47960    192.168.152.24:8220     ESTABLISHED 411726/elastic-agen  keepalive (2.86/0/0)
tcp6       0      0 :::8220                 :::*                    LISTEN      411753/fleet-server  off (0.00/0/0)
tcp6       0      0 192.168.152.24:8220     192.168.152.76:60600    ESTABLISHED 411753/fleet-server  keepalive (10.05/0/0)
tcp6       0      0 192.168.152.24:8220     192.168.152.24:47960    ESTABLISHED 411753/fleet-server  keepalive (2.86/0/0)
root@elastic3:~# ps ax | grep ap[m]
 411767 ?        Sl     0:00 /var/lib/elastic-agent/data/elastic-agent-6118f2/install/apm-server-8.1.2-linux-x86_64/apm-server -E management.enabled=true -E gc_percent=${APMSERVER_GOGC:100} -E logging.level=info -E http.enabled=true -E http.host=unix:///var/lib/elastic-agent/data/tmp/default/apm-server/apm-server.sock -E logging.files.path=/var/lib/elastic-agent/data/elastic-agent-6118f2/logs/default -E logging.files.name=apm-server -E logging.files.keepfiles=7 -E logging.files.permission=0640 -E logging.files.interval=1h -E path.data=/var/lib/elastic-agent/data/elastic-agent-6118f2/run/default/apm-server--8.1.2
root@elastic3:~#

dthapostle · April 22, 2022, 6:06am

@simitt Could you continue your assistance?

xeraa · April 26, 2022, 2:42am

What do you get when you ran netstat on the process ID of the APM-server (411767 in your example)?

dthapostle · April 26, 2022, 6:43am

Hi @xeraa.

root@elastic3:/home/aleksey# ps ax | grep ap[m]
 412657 ?        Sl    26:30 /var/lib/elastic-agent/data/elastic-agent-6118f2/install/apm-server-8.1.2-linux-x86_64/apm-server -E management.enabled=true -E gc_percent=${APMSERVER_GOGC:100} -E logging.level=info -E http.enabled=true -E http.host=unix:///var/lib/elastic-agent/data/tmp/default/apm-server/apm-server.sock -E logging.files.path=/var/lib/elastic-agent/data/elastic-agent-6118f2/logs/default -E logging.files.name=apm-server -E logging.files.keepfiles=7 -E logging.files.permission=0640 -E logging.files.interval=1h -E path.data=/var/lib/elastic-agent/data/elastic-agent-6118f2/run/default/apm-server--8.1.2
root@elastic3:/home/aleksey# netstat -anop | grep 412657
tcp        0      0 127.0.0.1:36322         127.0.0.1:6789          ESTABLISHED 412657/apm-server    keepalive (10.46/0/0)
unix  2      [ ACC ]     STREAM     LISTENING     962546   412657/apm-server    /var/lib/elastic-agent/data/tmp/default/apm-server/apm-server.sock
unix  3      [ ]         STREAM     CONNECTED     969762   412657/apm-server    /var/lib/elastic-agent/data/tmp/default/apm-server/apm-server.sock
unix  3      [ ]         STREAM     CONNECTED     969766   412657/apm-server    /var/lib/elastic-agent/data/tmp/default/apm-server/apm-server.sock
unix  3      [ ]         STREAM     CONNECTED     969764   412657/apm-server    /var/lib/elastic-agent/data/tmp/default/apm-server/apm-server.sock
root@elastic3:/home/aleksey#

simitt · April 26, 2022, 12:57pm

Can you spot anything helpful in the Elastic Agent or APM Server logs? You should be able to view them via Kibana or access them directly from the elastic3 container (see the default logging parameters).

xeraa · April 27, 2022, 2:05am

I don't have that on-prem setup running right now to check, but I'm surprised the only LISTENING is a unix socket (for the HTTP metrics) but nothing on port 8200 (or whatever you have configured) — unless that's what you'd expect, Silvia?

Otherwise the logs might be interesting around what this is binding to.

dthapostle · April 27, 2022, 7:42am

@simitt @xeraa
I uploaded logs to OneDrive

dthapostle · May 2, 2022, 11:06am

@simitt @xeraa any ideas?

system · May 23, 2022, 7:07am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.