APM host on Standalone Elastic Agent

Hello,

I'm using Standalone Elastic agent on version 7.17.7. Tested it on docker and kubernetes, but in both cases can get to connect my APM agent to Elastic's agent APM, always getting connection refused error. When going to elastic agent, apm logs shows service starts correctly at [::]:8200 but when executing telnet this port is nowhere to be found, as if its never opened.

I can see Elastic agent's 6789 port listening on localhost but nothing else related. How should i construct the APM server url to make it work with elastic agent and reachable from outside or even on localhost?

Thanks for your time

@juliana_cg are you able to share your Kubernetes manifests, so we can try to reproduce the issue?

Thanks for the reply!

Here are my kubernetes resources

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: agent-node-datastreams
  namespace: kube-elasticagent
  labels:
    k8s-app: elastic-agent
data:
  agent.yml: |-
    outputs:
      default:
        type: elasticsearch
        hosts:
          - >-
            ${ES_HOST}
        username: ${ES_USERNAME}
        password: ${ES_PASSWORD}
        ssl:
          verification_mode: none
    output_permissions:
      default:
        _elastic_agent_monitoring:
          indices:
            - names:
                - logs-elastic_agent.cloudbeat-test_ns
              privileges: &ref_0
                - auto_configure
                - create_doc
            - names:
                - logs-elastic_agent.apm_server-test_ns
              privileges: *ref_0
            - names:
                - logs-elastic_agent.auditbeat-test_ns
              privileges: *ref_0
            - names:
                - metrics-elastic_agent.cloudbeat-test_ns
              privileges: *ref_0
            - names:
                - logs-elastic_agent-test_ns
              privileges: *ref_0
            - names:
                - metrics-elastic_agent.apm_server-test_ns
              privileges: *ref_0
            - names:
                - metrics-elastic_agent.auditbeat-test_ns
              privileges: *ref_0
            - names:
                - metrics-elastic_agent.metricbeat-test_ns
              privileges: *ref_0
            - names:
                - metrics-elastic_agent.filebeat-test_ns
              privileges: *ref_0
            - names:
                - logs-elastic_agent.metricbeat-test_ns
              privileges: *ref_0
            - names:
                - metrics-elastic_agent.elastic_agent-test_ns
              privileges: *ref_0
            - names:
                - metrics-elastic_agent.osquerybeat-test_ns
              privileges: *ref_0
            - names:
                - logs-elastic_agent.packetbeat-test_ns
              privileges: *ref_0
            - names:
                - metrics-elastic_agent.packetbeat-test_ns
              privileges: *ref_0
            - names:
                - logs-elastic_agent.osquerybeat-test_ns
              privileges: *ref_0
            - names:
                - metrics-elastic_agent.endpoint_security-test_ns
              privileges: *ref_0
            - names:
                - logs-elastic_agent.filebeat-test_ns
              privileges: *ref_0
            - names:
                - logs-elastic_agent.heartbeat-test_ns
              privileges: *ref_0
            - names:
                - metrics-elastic_agent.fleet_server-test_ns
              privileges: *ref_0
            - names:
                - logs-elastic_agent.fleet_server-test_ns
              privileges: *ref_0
            - names:
                - metrics-elastic_agent.heartbeat-test_ns
              privileges: *ref_0
            - names:
                - logs-elastic_agent.endpoint_security-test_ns
              privileges: *ref_0
        _elastic_agent_checks:
          cluster:
            - monitor
        apm-2:
          indices:
            - names:
                - logs-apm.app-test_ns
              privileges: *ref_0
            - names:
                - logs-apm.error-test_ns
              privileges: *ref_0
            - names:
                - traces-apm.sampled-test_ns
              privileges:
                - auto_configure
                - create_doc
                - maintenance
                - monitor
                - read
            - names:
                - metrics-apm.profiling-test_ns
              privileges: *ref_0
            - names:
                - traces-apm-test_ns
              privileges: *ref_0
            - names:
                - metrics-apm.app.*-test_ns
              privileges: *ref_0
            - names:
                - metrics-apm.internal-test_ns
              privileges: *ref_0
          cluster:
            - 'cluster:monitor/main'
        system-metrics:
          indices:
            - names:
                - metrics-system.fsstat-test_ns
              privileges: *ref_0
            - names:
                - metrics-system.network-test_ns
              privileges: *ref_0
            - names:
                - metrics-system.memory-test_ns
              privileges: *ref_0
            - names:
                - metrics-system.core-test_ns
              privileges: *ref_0
            - names:
                - metrics-system.cpu-test_ns
              privileges: *ref_0
            - names:
                - metrics-system.process.summary-test_ns
              privileges: *ref_0
            - names:
                - metrics-system.uptime-test_ns
              privileges: *ref_0
            - names:
                - metrics-system.filesystem-test_ns
              privileges: *ref_0
            - names:
                - metrics-system.diskio-test_ns
              privileges: *ref_0
            - names:
                - metrics-system.socket_summary-test_ns
              privileges: *ref_0
            - names:
                - metrics-system.process-test_ns
              privileges: *ref_0
            - names:
                - metrics-system.load-test_ns
              privileges: *ref_0
    agent:
      logging:
        level: debug
      monitoring:
        enabled: true
        use_output: default
        namespace: test_ns
        logs: true
        metrics: true
    providers:
      kubernetes:
        node: ${NODE_NAME}
        scope: node
      kubernetes_leaderelection:
        enabled: false
    inputs:
      - name: apm-2
        revision: 2
        type: apm
        use_output: default
        meta:
          package:
            name: apm
            version: 7.17.0
        data_stream:
          namespace: test_ns
        apm-server:
          capture_personal_data: true
          max_connections: 0
          max_event_size: 307200
          auth:
            api_key:
              enabled: false
              limit: 100
            anonymous:
              enabled: true
              allow_agent:
                - rum-js
                - js-base
                - iOS/swift
              allow_service: null
              rate_limit:
                ip_limit: 1000
                event_limit: 300
            secret_token: null
          default_service_environment: null
          shutdown_timeout: 30s
          sampling:
            tail:
              enabled: false
              policies: null
          rum:
            enabled: true
            exclude_from_grouping: ^/webpack
            allow_headers: null
            response_headers: null
            library_pattern: node_modules|bower_components|~
            allow_origins:
              - '*'
            source_mapping:
              metadata: []
          ssl:
            enabled: false
            key_passphrase: null
            certificate: null
            supported_protocols:
              - TLSv1.0
              - TLSv1.1
              - TLSv1.2
            curve_types: null
            key: null
            cipher_suites: null
          response_headers: null
          write_timeout: 30s
          host: "0.0.0.0:8200"
          max_header_size: 1048576
          idle_timeout: 45s
          expvar.enabled: false
          read_timeout: 3600s
          java_attacher:
            enabled: null
            discovery-rules: null
            download-agent-version: null
          agent_config:
            - service: {}
              etag: b52d6aa9cbcb5508c6edfb57cf7770090856f77f
              config:
                transaction_max_spans: '500'
                transaction_sample_rate: '1.0'
      - name: system-metrics
        type: system/metrics
        use_output: default
        meta:
          package:
            name: system
            version: 0.11.0
        data_stream:
          namespace: test_ns
        streams:
          - data_stream:
              dataset: system.core
              type: metrics
            metricsets:
              - core
            core.metrics:
              - percentages
          - data_stream:
              dataset: system.cpu
              type: metrics
            period: 10s
            cpu.metrics:
              - percentages
              - normalized_percentages
            metricsets:
              - cpu
          - data_stream:
              dataset: system.diskio
              type: metrics
            period: 10s
            diskio.include_devices: null
            metricsets:
              - diskio
          - data_stream:
              dataset: system.filesystem
              type: metrics
            period: 1m
            metricsets:
              - filesystem
            processors:
              - drop_event.when.regexp:
                  system.filesystem.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
          - data_stream:
              dataset: system.fsstat
              type: metrics
            period: 1m
            metricsets:
              - fsstat
            processors:
              - drop_event.when.regexp:
                  system.fsstat.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
          - data_stream:
              dataset: system.load
              type: metrics
            period: 10s
            metricsets:
              - load
          - data_stream:
              dataset: system.memory
              type: metrics
            period: 10s
            metricsets:
              - memory
          - data_stream:
              dataset: system.network
              type: metrics
            period: 10s
            network.interfaces: null
            metricsets:
              - network
          - data_stream:
              dataset: system.process
              type: metrics
            process.include_top_n.by_memory: 5
            period: 10s
            processes:
              - .*
            process.include_top_n.by_cpu: 5
            process.cgroups.enabled: false
            process.cmdline.cache.enabled: true
            metricsets:
              - process
            process.include_cpu_ticks: false
            system.hostfs: /hostfs
          - data_stream:
              dataset: system.process_summary
              type: metrics
            period: 10s
            metricsets:
              - process_summary
            system.hostfs: /hostfs
          - data_stream:
              dataset: system.socket_summary
              type: metrics
            period: 10s
            metricsets:
              - socket_summary
            system.hostfs: /hostfs            
---

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: elastic-agent
  namespace: kube-elasticagent
  labels:
    app: elastic-agent
spec:
  selector:
    matchLabels:
      app: elastic-agent
  template:
    metadata:
      labels:
        app: elastic-agent
    spec:
      serviceAccountName: elastic-agent
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      containers:
        - name: elastic-agent
          image: docker.elastic.co/beats/elastic-agent:7.17.6
          args: [
            "-c", "/etc/agent.yml",
            "-e",
          ]
          env:
            - name: ES_USERNAME
              value: "elastic"
            - name: ES_PASSWORD
              value: "changeme"
            - name: ES_HOST
              value: "elasticurl"
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: NODE_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP
          securityContext:
            runAsUser: 0
          ports:
            - name: apm
              containerPort: 8200
              hostPort: 8200
          resources:
            limits:
              memory: 500Mi
            requests:
              cpu: 100m
              memory: 200Mi
          volumeMounts:
            - name: datastreams
              mountPath: /etc/agent.yml
              readOnly: true
              subPath: agent.yml
            - name: proc
              mountPath: /hostfs/proc
              readOnly: true
            - name: cgroup
              mountPath: /hostfs/sys/fs/cgroup
              readOnly: true
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: varlog
              mountPath: /var/log
              readOnly: true
      volumes:
        - name: datastreams
          configMap:
            defaultMode: 0640
            name: agent-node-datastreams
        - name: proc
          hostPath:
            path: /proc
        - name: cgroup
          hostPath:
            path: /sys/fs/cgroup
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: varlog
          hostPath:
            path: /var/log
---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: elastic-agent
subjects:
  - kind: ServiceAccount
    name: elastic-agent
    namespace: kube-elasticagent
roleRef:
  kind: ClusterRole
  name: elastic-agent
  apiGroup: rbac.authorization.k8s.io
---

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: kube-elasticagent
  name: elastic-agent
subjects:
  - kind: ServiceAccount
    name: elastic-agent
    namespace: kube-elasticagent
roleRef:
  kind: Role
  name: elastic-agent
  apiGroup: rbac.authorization.k8s.io
---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: elastic-agent
  labels:
    k8s-app: elastic-agent
rules:
  - apiGroups: [""]
    resources:
      - nodes
      - namespaces
      - events
      - pods
      - services
      - configmaps
    verbs: ["get", "list", "watch"]
  - apiGroups: ["extensions"]
    resources:
      - replicasets
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources:
      - statefulsets
      - deployments
      - replicasets
    verbs: ["get", "list", "watch"]
  - apiGroups: ["batch"]
    resources:
      - jobs
    verbs: ["get", "list", "watch"]
  - apiGroups:
      - ""
    resources:
      - nodes/stats
    verbs:
      - get
  # required for apiserver
  - nonResourceURLs:
      - "/metrics"
    verbs:
      - get
---

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: elastic-agent
  # should be the namespace where elastic-agent is running
  namespace: kube-elasticagent
  labels:
    k8s-app: elastic-agent
rules:
  - apiGroups:
      - coordination.k8s.io
    resources:
      - leases
    verbs: ["get", "create", "update"]
  - apiGroups: ['policy']
    resources: ['podsecuritypolicies']
    verbs:     ['use']
    resourceNames:
    - elastic-agent
---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: elastic-agent
  namespace: kube-elasticagent
  labels:
    k8s-app: elastic-agent
---

apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
  name: elastic-agent
spec:
  allowPrivilegeEscalation: false
  allowedHostPaths:
  - pathPrefix: /var/lib/docker/containers
    readOnly: true
  - pathPrefix: /var/log
    readOnly: true
  - pathPrefix: /proc
    readOnly: true
  - pathPrefix: /sys/fs/cgroup
    readOnly: true
  fsGroup:
    rule: RunAsAny
  runAsUser:
    rule: RunAsAny
  seLinux:
    rule: RunAsAny
  supplementalGroups:
    rule: RunAsAny
  volumes:
  - configMap
  - secret
  - hostPath
  - emptyDir
  hostNetwork: true
  hostPorts:
  - min: 8200
    max: 8200

I got same behaviour on docker, so here are my docker files, in case is easier to test.

docker-compose.yml

version: "3"
services:
  elastic-agent:
    image: docker.elastic.co/beats/elastic-agent:7.17.7
    container_name: elastic-agent
    restart: always
    user: root # note, synthetic browser monitors require this set to `elastic-agent`
    volumes:
      - ./elastic-agent.yml:/usr/share/elastic-agent/elastic-agent.yml
    ports:
      - 8200:8200 

elastic-agent.yml

###################### Agent Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The elastic-agent.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.

######################################
# Fleet configuration
######################################
outputs:
  default:
    type: elasticsearch
    hosts: [elasticurl]
    username: elastic
    password: changeme
    ssl.verification_mode: none

inputs:
  - type: system/metrics

    # Namespace name must conform to the naming conventions for Elasticsearch indices, cannot contain dashes (-), and cannot exceed 100 bytes
    # For index naming restrictions, see https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#indices-create-api-path-params
    data_stream.namespace: local_test
    use_output: default
    streams:
      - metricset: cpu
        # Dataset name must conform to the naming conventions for Elasticsearch indices, cannot contain dashes (-), and cannot exceed 100 bytes
        data_stream.dataset: system.cpu
      - metricset: memory
        data_stream.dataset: system.memory
      - metricset: network
        data_stream.dataset: system.network
      - metricset: filesystem
        data_stream.dataset: system.filesystem

  - name: apm-2
    revision: 2
    type: apm
    use_output: default
    meta:
      package:
        name: apm
        version: 7.17.0
    data_stream:
      namespace: local_test
    apm-server:
      capture_personal_data: true
      max_connections: 0
      max_event_size: 307200
      auth:
        api_key:
          enabled: true
          limit: 100
        anonymous:
          enabled: true
          allow_agent:
            - rum-js
            - js-base
            - iOS/swift
          allow_service: null
          rate_limit:
            ip_limit: 1000
            event_limit: 300
        secret_token: abcd1234
      default_service_environment: null
      shutdown_timeout: 30s
      sampling:
        tail:
          enabled: false
          policies: null
      rum:
        enabled: true
        exclude_from_grouping: ^/webpack
        allow_headers: null
        response_headers: null
        library_pattern: node_modules|bower_components|~
        allow_origins:
          - '*'
        source_mapping:
          metadata: []
      ssl:
        enabled: false
        key_passphrase: null
        certificate: null
        supported_protocols:
          - TLSv1.0
          - TLSv1.1
          - TLSv1.2
        curve_types: null
        key: null
        cipher_suites: null
      response_headers: null
      write_timeout: 30s
      host: "0.0.0.0:8200"
      max_header_size: 1048576
      idle_timeout: 45s
      expvar.enabled: false
      read_timeout: 3600s
      java_attacher:
        enabled: null
        discovery-rules: null
        download-agent-version: null
      agent_config:
        - service: {}
          config:
            transaction_max_spans: '500'
            transaction_sample_rate: '1.0'

# Send all logging output to stderr. The default is false.
agent.logging.to_stderr: true

Elastic url, username, and password are set up correctly. I get system metrics on my elastic so metricbeats does work fine.

Let me know if i can provide any more info.

@juliana_cg I can reproduce the symptoms you're seeing. I haven't gotten to the bottom of why this is happening; there's likely some assumptions built in about Fleet.

In case you missed it, APM Server doesn't currently support Elastic Agent standalone, as mentioned in a note at Quick start | APM User Guide [8.5] | Elastic

We are thinking about making some changes to enable that. In the mean time, is running Legacy APM Server an option for you? Getting started with APM Server | APM User Guide [8.5] | Elastic

Hey @axw

In case you missed it, APM Server doesn't currently support Elastic Agent standalone, as mentioned in a note at Quick start | APM User Guide [8.5] | Elastic

Totally missed this :confused: it may not be present on 7.17 docs

In the mean time, is running Legacy APM Server an option for you?

I'm not sure, i get this on APM's app settings

So I assumed I couldn't use legacy apm anymore, is this correct?

I'm not sure, i get this on APM's app settings

Ah, I see. If you have migrated to data streams, then you should really be setting up Elastic Agent with Fleet. However...

So I assumed I couldn't use legacy apm anymore, is this correct?

In 7.17 you can configure legacy APM Server to write to data streams by setting apm-server.data_streams.enabled: true in apm-server.yml. This is the default in 8.0+

1 Like

Oh, i see. That's great news. Will try to deploy apm server on kubernetes then. Thanks a lot!