Is standalone APM server required with Elastic APM Integration?

[pokaleshrey (Shreyash Pokale)]

Hi,
I have installed ES, Kibana. And its up and running.

Further, i have added Elastic APM Integration into Kibana, and have setup Fleet server, Elastic Agent.

(note i did not setup any APM server yet)

I am trying to connect to http://localhost:8200 using APM Agent jar. But i get below error:

Elastic APM server http://localhost:8200/ is not available (Connection refused: connect)

Also if i telnet localhost 8200, it does not result into a connection.

Query: When i do a setup as above using APM Integration (recommended) approach do i need to setup APM Server externally or the above setup provides an internal APM server ??

[Eyal_Koren]

Elastic Agent would set up a local APM Server if it is enrolled with to a policy that contains the APM integration, so localhost:8200 should be valid on such hosts.

If you go to Management -> Fleet, you can observe agents, policies and other settings. In the Agents tab, you should see a list of your agents. Do you see your agent? Does it have a "Healthy" status or other? You can get further info on what's going on if you click on the agent and switch to the Logs tab, where you can see all agent-related logs. By default, only the root agent logs are shown, so you should mark the apm_server dataset in the Dataset dropdown.
Otherwise, you can login into the host and collect logs from the agent installation folder.
Hopefully this is at least somewhat helpful

[pokaleshrey (Shreyash Pokale)]

Yes my Agent is enrolled and showing as Healthy !

Though, i cannot find an option "apm_server" under Dataset drop down

Does it mean that apm_server was not installed ?

[Eyal_Koren]

Could be, although the Dataset selection may not be working in some versions.
If you see other datasets related to other integrations (e.g. metricbeat or filebeat), then I assume it does mean it is not installed, in which case you should verify that the APM integration is indeed configured within the policy that is applied to this agent and if so, look within the elastic_agent dataset and look for hints why it failed to start an APM Server.
If you only see the elastic_agent dataset although other integrations are configured, you should assume this is a bug in the UI (already fixed for upcoming version AFAIK) and you should log into the host and look for the logs in the agent installation dir.

BTW, what OS is it?

[pokaleshrey (Shreyash Pokale)]

I tried to look into elastic_agent logs and found no where abouts of "apm" word apart from below:

elastic_agent       [elastic_agent][info] APM instrumentation disabled

Drop down contains elastic_agent, filebeat, fleet_server & metricbeat.

Os: Windows

Tried to look into the host where elastic_agent is installed, but unfortunately its the same logs i see in the kibana

Also check in logs at \Elastic\Agent\data\elastic-agent-16c55b\logs\default. It does not even mentioned about apm-server

[pokaleshrey (Shreyash Pokale)]

Elastic Agent is somehow not able to understand that it has to install the apm-server too. It has downloaded it in its data folder but not installed it, like it did for fleet, metricbeat, filebeat.

I have installed ES and Kibana as standalone on my local machine. Rest i have installed using kibana UI. The sequence of installation was ElasticAPM -> FleetServer -> ElasticAgent.

[Eyal_Koren]

All I can think of is that the APM integration is not enabled. Please verify that your policy has APM integration enabled and configured to use localhost:8200.
Also verify that Fleet host and Elasticsearch endpoint is properly configured in Management -> Fleet -> Settings

If you still can't figure this out, try to share your policy.

[pokaleshrey (Shreyash Pokale)]

Just checked, things seem to be fine to me. May be it needs an expert eye.

Below is the fleet host configuration and my agent policy.

id: 09aa5220-0ce3-11ed-aa5f-51a843e8c7f2
revision: 5
outputs:
  default:
    type: elasticsearch
    hosts:
      - 'https://<host-ip>:9200'
    ssl.ca_trusted_fingerprint: xxxx
output_permissions:
  default:
    _elastic_agent_monitoring:
      indices:
        - names:
            - logs-elastic_agent.apm_server-default
          privileges: &ref_0
            - auto_configure
            - create_doc
        - names:
            - metrics-elastic_agent.apm_server-default
          privileges: *ref_0
        - names:
            - logs-elastic_agent.auditbeat-default
          privileges: *ref_0
        - names:
            - metrics-elastic_agent.auditbeat-default
          privileges: *ref_0
        - names:
            - logs-elastic_agent.cloudbeat-default
          privileges: *ref_0
        - names:
            - metrics-elastic_agent.cloudbeat-default
          privileges: *ref_0
        - names:
            - logs-elastic_agent-default
          privileges: *ref_0
        - names:
            - metrics-elastic_agent.elastic_agent-default
          privileges: *ref_0
        - names:
            - metrics-elastic_agent.endpoint_security-default
          privileges: *ref_0
        - names:
            - logs-elastic_agent.endpoint_security-default
          privileges: *ref_0
        - names:
            - logs-elastic_agent.filebeat-default
          privileges: *ref_0
        - names:
            - metrics-elastic_agent.filebeat-default
          privileges: *ref_0
        - names:
            - logs-elastic_agent.fleet_server-default
          privileges: *ref_0
        - names:
            - metrics-elastic_agent.fleet_server-default
          privileges: *ref_0
        - names:
            - logs-elastic_agent.heartbeat-default
          privileges: *ref_0
        - names:
            - metrics-elastic_agent.heartbeat-default
          privileges: *ref_0
        - names:
            - logs-elastic_agent.metricbeat-default
          privileges: *ref_0
        - names:
            - metrics-elastic_agent.metricbeat-default
          privileges: *ref_0
        - names:
            - logs-elastic_agent.osquerybeat-default
          privileges: *ref_0
        - names:
            - metrics-elastic_agent.osquerybeat-default
          privileges: *ref_0
        - names:
            - logs-elastic_agent.packetbeat-default
          privileges: *ref_0
        - names:
            - metrics-elastic_agent.packetbeat-default
          privileges: *ref_0
    _elastic_agent_checks:
      cluster:
        - monitor
    de8ad43c-1296-4573-a928-0f0ec2b24518:
      indices:
        - names:
            - logs-system.auth-default
          privileges: *ref_0
        - names:
            - logs-system.syslog-default
          privileges: *ref_0
        - names:
            - logs-system.application-default
          privileges: *ref_0
        - names:
            - logs-system.security-default
          privileges: *ref_0
        - names:
            - logs-system.system-default
          privileges: *ref_0
        - names:
            - metrics-system.cpu-default
          privileges: *ref_0
        - names:
            - metrics-system.diskio-default
          privileges: *ref_0
        - names:
            - metrics-system.filesystem-default
          privileges: *ref_0
        - names:
            - metrics-system.fsstat-default
          privileges: *ref_0
        - names:
            - metrics-system.load-default
          privileges: *ref_0
        - names:
            - metrics-system.memory-default
          privileges: *ref_0
        - names:
            - metrics-system.network-default
          privileges: *ref_0
        - names:
            - metrics-system.process-default
          privileges: *ref_0
        - names:
            - metrics-system.process.summary-default
          privileges: *ref_0
        - names:
            - metrics-system.socket_summary-default
          privileges: *ref_0
        - names:
            - metrics-system.uptime-default
          privileges: *ref_0
    f5116873-86eb-4907-a0dc-5df9c7ae3acc:
      indices:
        - names:
            - logs-apm.app-default
          privileges: *ref_0
        - names:
            - metrics-apm.app.*-default
          privileges: *ref_0
        - names:
            - logs-apm.error-default
          privileges: *ref_0
        - names:
            - metrics-apm.internal-default
          privileges: *ref_0
        - names:
            - metrics-apm.profiling-default
          privileges: *ref_0
        - names:
            - traces-apm.rum-default
          privileges: *ref_0
        - names:
            - traces-apm.sampled-default
          privileges:
            - auto_configure
            - create_doc
            - maintenance
            - monitor
            - read
        - names:
            - traces-apm-default
          privileges: *ref_0
agent:
  monitoring:
    enabled: true
    use_output: default
    namespace: default
    logs: true
    metrics: true
inputs:
  - id: logfile-system-de8ad43c-1296-4573-a928-0f0ec2b24518
    name: system-1
    revision: 1
    type: logfile
    use_output: default
    meta:
      package:
        name: system
        version: 1.16.2
    data_stream:
      namespace: default
    streams:
      - id: logfile-system.auth-de8ad43c-1296-4573-a928-0f0ec2b24518
        data_stream:
          dataset: system.auth
          type: logs
        paths:
          - /var/log/auth.log*
          - /var/log/secure*
        exclude_files:
          - .gz$
        multiline:
          pattern: ^\s
          match: after
        processors:
          - add_locale: null
      - id: logfile-system.syslog-de8ad43c-1296-4573-a928-0f0ec2b24518
        data_stream:
          dataset: system.syslog
          type: logs
        paths:
          - /var/log/messages*
          - /var/log/syslog*
        exclude_files:
          - .gz$
        multiline:
          pattern: ^\s
          match: after
        processors:
          - add_locale: null
  - id: winlog-system-de8ad43c-1296-4573-a928-0f0ec2b24518
    name: system-1
    revision: 1
    type: winlog
    use_output: default
    meta:
      package:
        name: system
        version: 1.16.2
    data_stream:
      namespace: default
    streams:
      - id: winlog-system.application-de8ad43c-1296-4573-a928-0f0ec2b24518
        name: Application
        data_stream:
          dataset: system.application
          type: logs
        condition: '${host.platform} == ''windows'''
        ignore_older: 72h
      - id: winlog-system.security-de8ad43c-1296-4573-a928-0f0ec2b24518
        name: Security
        data_stream:
          dataset: system.security
          type: logs
        condition: '${host.platform} == ''windows'''
        ignore_older: 72h
      - id: winlog-system.system-de8ad43c-1296-4573-a928-0f0ec2b24518
        name: System
        data_stream:
          dataset: system.system
          type: logs
        condition: '${host.platform} == ''windows'''
        ignore_older: 72h
  - id: system/metrics-system-de8ad43c-1296-4573-a928-0f0ec2b24518
    name: system-1
    revision: 1
    type: system/metrics
    use_output: default
    meta:
      package:
        name: system
        version: 1.16.2
    data_stream:
      namespace: default
    streams:
      - id: system/metrics-system.cpu-de8ad43c-1296-4573-a928-0f0ec2b24518
        data_stream:
          dataset: system.cpu
          type: metrics
        metricsets:
          - cpu
        cpu.metrics:
          - percentages
          - normalized_percentages
        period: 10s
      - id: system/metrics-system.diskio-de8ad43c-1296-4573-a928-0f0ec2b24518
        data_stream:
          dataset: system.diskio
          type: metrics
        metricsets:
          - diskio
        diskio.include_devices: null
        period: 10s
      - id: system/metrics-system.filesystem-de8ad43c-1296-4573-a928-0f0ec2b24518
        data_stream:
          dataset: system.filesystem
          type: metrics
        metricsets:
          - filesystem
        period: 1m
        processors:
          - drop_event.when.regexp:
              system.filesystem.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
      - id: system/metrics-system.fsstat-de8ad43c-1296-4573-a928-0f0ec2b24518
        data_stream:
          dataset: system.fsstat
          type: metrics
        metricsets:
          - fsstat
        period: 1m
        processors:
          - drop_event.when.regexp:
              system.fsstat.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
      - id: system/metrics-system.load-de8ad43c-1296-4573-a928-0f0ec2b24518
        data_stream:
          dataset: system.load
          type: metrics
        metricsets:
          - load
        condition: '${host.platform} != ''windows'''
        period: 10s
      - id: system/metrics-system.memory-de8ad43c-1296-4573-a928-0f0ec2b24518
        data_stream:
          dataset: system.memory
          type: metrics
        metricsets:
          - memory
        period: 10s
      - id: system/metrics-system.network-de8ad43c-1296-4573-a928-0f0ec2b24518
        data_stream:
          dataset: system.network
          type: metrics
        metricsets:
          - network
        period: 10s
        network.interfaces: null
      - id: system/metrics-system.process-de8ad43c-1296-4573-a928-0f0ec2b24518
        data_stream:
          dataset: system.process
          type: metrics
        metricsets:
          - process
        period: 10s
        process.include_top_n.by_cpu: 5
        process.include_top_n.by_memory: 5
        process.cmdline.cache.enabled: true
        process.cgroups.enabled: false
        process.include_cpu_ticks: false
        processes:
          - .*
      - id: >-
          system/metrics-system.process.summary-de8ad43c-1296-4573-a928-0f0ec2b24518
        data_stream:
          dataset: system.process.summary
          type: metrics
        metricsets:
          - process_summary
        period: 10s
      - id: >-
          system/metrics-system.socket_summary-de8ad43c-1296-4573-a928-0f0ec2b24518
        data_stream:
          dataset: system.socket_summary
          type: metrics
        metricsets:
          - socket_summary
        period: 10s
      - id: system/metrics-system.uptime-de8ad43c-1296-4573-a928-0f0ec2b24518
        data_stream:
          dataset: system.uptime
          type: metrics
        metricsets:
          - uptime
        period: 10s
  - id: fleet-server-fleet_server-ddc0c3a0-2e57-4728-9113-ec29bacd5b67
    name: fleet_server-1
    revision: 1
    type: fleet-server
    use_output: default
    meta:
      package:
        name: fleet_server
        version: 1.2.0
    data_stream:
      namespace: default
    server:
      port: 8220
      host: 0.0.0.0
  - id: f5116873-86eb-4907-a0dc-5df9c7ae3acc
    name: apm-1
    revision: 1
    type: apm
    use_output: default
    meta:
      package:
        name: apm
        version: 8.3.0
    data_stream:
      namespace: default
    apm-server:
      auth:
        anonymous:
          enabled: true
          allow_agent:
            - rum-js
            - js-base
            - iOS/swift
          allow_service: null
          rate_limit:
            event_limit: 300
            ip_limit: 1000
        api_key:
          enabled: false
          limit: 100
        secret_token: null
      capture_personal_data: true
      idle_timeout: 45s
      default_service_environment: null
      expvar.enabled: false
      pprof.enabled: false
      host: 'localhost:8200'
      max_connections: 0
      max_event_size: 307200
      max_header_size: 1048576
      read_timeout: 3600s
      response_headers: null
      java_attacher:
        enabled: false
        discovery-rules: null
        download-agent-version: null
      rum:
        enabled: true
        allow_headers: null
        allow_origins:
          - '*'
        exclude_from_grouping: ^/webpack
        library_pattern: node_modules|bower_components|~
        response_headers: null
        source_mapping:
          metadata: []
      shutdown_timeout: 30s
      ssl:
        enabled: false
        certificate: null
        key: null
        key_passphrase: null
        supported_protocols:
          - TLSv1.0
          - TLSv1.1
          - TLSv1.2
        cipher_suites: null
        curve_types: null
      write_timeout: 30s
      sampling:
        tail:
          enabled: false
          interval: 1m
          policies:
            - sample_rate: 0.1
      agent_config: []
fleet:
  hosts:
    - 'https://localhost:8220'

[Eyal_Koren]

@pokaleshrey

May be it needs an expert eye

Right, unfortunately I am no such either
Tagging as Fleet and APM Server related, hopefully someone can tell why your APM Server doesn't even attempts to start.

[pokaleshrey (Shreyash Pokale)]

Ok. No problem. Thanks for extending the tags.

Hopefully someone can help with apm-server start as a part of elastic-agent.

[Ivan.fernandez]

Have your policy the APM integration installed and configured? It should look like this one.

you can configure it directly on your kibana.yml file with the following settings

xpack.fleet.agents.enabled: true
xpack.fleet.packages:
    - name: system
      version: latest
    - name: elastic_agent
      version: latest
    - name: apm
      version: latest
    - name: fleet_server
      version: latest

xpack.fleet.agentPolicies:
    - name: Fleet Server + APM policy
      id: fleet-server-apm-policy
      description: Fleet server policy with APM and System logs and metrics enabled
      namespace: default
      is_default_fleet_server: true
      is_managed: false
      monitoring_enabled:
          - logs
          - metrics
      package_policies:
          - name: apm-1
            id: default-apm
            package:
                name: apm
            inputs:
                - type: apm
                  keep_enabled: true
                  vars:
                      - name: host
                        value: 0.0.0.0:8200
                        frozen: true
                      - name: url
                        value: http://my-host-url:8200
                        frozen: true
          - name: Fleet Server
            package:
                name: fleet_server
            inputs:
                - type: fleet-server
                  keep_enabled: true
                  vars:
                      - name: host
                        value: 0.0.0.0
                        frozen: true
                      - name: port
                        value: 8220
                        frozen: true

[pokaleshrey (Shreyash Pokale)]

Yup. Indeed the agent policy does contain the APM Integration, same as the screenshot you have posted

[Ivan.fernandez]

Which host and URL do you have configured? How you are running the Elastic Stack and Elastic Agent (Docker, Docker compose, K8s, VMs,...)?

[pokaleshrey (Shreyash Pokale)]

OS: Windows
ElasticSearch and Kibana are extracted binaries on my local machine.
ElasticAgent is running as a service on my local windows machine.
I wanted to give it a quick try hence have not used any of the docker, K8s or a VM deployment yet.

So, in all, its all localhost. Below is the screenshot of my Elastic APM integration:

[Ivan.fernandez]

If your agent is up and running and appears as healthy in fleet agents, it should accept connections on localhost with your configuration. However, I dunno the configuration in your windows, and I dunno if the loopback network interface is running, so I will suggest using "host: 0.0.0.0:8200" and "URL: HTTP://YOUR_WINDOWS_EXTERNAL_IP:8200"

[pokaleshrey (Shreyash Pokale)]

Ok. But i cannot see an apm-server installation inside my agent. Although agent has downloaded the apm-server setup in the \Elastic\Agent\data\elastic-agent-16c55b\downloads

Below is a screenshot of what agent has installed within itself:

[Ivan.fernandez]

Could you stop the Elastic Agent service and run it from the command line with the following command? It should be errors in logs related to APM.

cd elastic-agent-install-folder
elastic-agent.exe -e -d "*"