Hello,
I'm using Standalone Elastic agent on version 7.17.7. Tested it on docker and kubernetes, but in both cases can get to connect my APM agent to Elastic's agent APM, always getting connection refused error. When going to elastic agent, apm logs shows service starts correctly at [::]:8200 but when executing telnet this port is nowhere to be found, as if its never opened.
I can see Elastic agent's 6789 port listening on localhost but nothing else related. How should i construct the APM server url to make it work with elastic agent and reachable from outside or even on localhost?
Thanks for your time
axw
November 1, 2022, 1:11am
2
@juliana_cg are you able to share your Kubernetes manifests, so we can try to reproduce the issue?
Thanks for the reply!
Here are my kubernetes resources
---
apiVersion: v1
kind: ConfigMap
metadata:
name: agent-node-datastreams
namespace: kube-elasticagent
labels:
k8s-app: elastic-agent
data:
agent.yml: |-
outputs:
default:
type: elasticsearch
hosts:
- >-
${ES_HOST}
username: ${ES_USERNAME}
password: ${ES_PASSWORD}
ssl:
verification_mode: none
output_permissions:
default:
_elastic_agent_monitoring:
indices:
- names:
- logs-elastic_agent.cloudbeat-test_ns
privileges: &ref_0
- auto_configure
- create_doc
- names:
- logs-elastic_agent.apm_server-test_ns
privileges: *ref_0
- names:
- logs-elastic_agent.auditbeat-test_ns
privileges: *ref_0
- names:
- metrics-elastic_agent.cloudbeat-test_ns
privileges: *ref_0
- names:
- logs-elastic_agent-test_ns
privileges: *ref_0
- names:
- metrics-elastic_agent.apm_server-test_ns
privileges: *ref_0
- names:
- metrics-elastic_agent.auditbeat-test_ns
privileges: *ref_0
- names:
- metrics-elastic_agent.metricbeat-test_ns
privileges: *ref_0
- names:
- metrics-elastic_agent.filebeat-test_ns
privileges: *ref_0
- names:
- logs-elastic_agent.metricbeat-test_ns
privileges: *ref_0
- names:
- metrics-elastic_agent.elastic_agent-test_ns
privileges: *ref_0
- names:
- metrics-elastic_agent.osquerybeat-test_ns
privileges: *ref_0
- names:
- logs-elastic_agent.packetbeat-test_ns
privileges: *ref_0
- names:
- metrics-elastic_agent.packetbeat-test_ns
privileges: *ref_0
- names:
- logs-elastic_agent.osquerybeat-test_ns
privileges: *ref_0
- names:
- metrics-elastic_agent.endpoint_security-test_ns
privileges: *ref_0
- names:
- logs-elastic_agent.filebeat-test_ns
privileges: *ref_0
- names:
- logs-elastic_agent.heartbeat-test_ns
privileges: *ref_0
- names:
- metrics-elastic_agent.fleet_server-test_ns
privileges: *ref_0
- names:
- logs-elastic_agent.fleet_server-test_ns
privileges: *ref_0
- names:
- metrics-elastic_agent.heartbeat-test_ns
privileges: *ref_0
- names:
- logs-elastic_agent.endpoint_security-test_ns
privileges: *ref_0
_elastic_agent_checks:
cluster:
- monitor
apm-2:
indices:
- names:
- logs-apm.app-test_ns
privileges: *ref_0
- names:
- logs-apm.error-test_ns
privileges: *ref_0
- names:
- traces-apm.sampled-test_ns
privileges:
- auto_configure
- create_doc
- maintenance
- monitor
- read
- names:
- metrics-apm.profiling-test_ns
privileges: *ref_0
- names:
- traces-apm-test_ns
privileges: *ref_0
- names:
- metrics-apm.app.*-test_ns
privileges: *ref_0
- names:
- metrics-apm.internal-test_ns
privileges: *ref_0
cluster:
- 'cluster:monitor/main'
system-metrics:
indices:
- names:
- metrics-system.fsstat-test_ns
privileges: *ref_0
- names:
- metrics-system.network-test_ns
privileges: *ref_0
- names:
- metrics-system.memory-test_ns
privileges: *ref_0
- names:
- metrics-system.core-test_ns
privileges: *ref_0
- names:
- metrics-system.cpu-test_ns
privileges: *ref_0
- names:
- metrics-system.process.summary-test_ns
privileges: *ref_0
- names:
- metrics-system.uptime-test_ns
privileges: *ref_0
- names:
- metrics-system.filesystem-test_ns
privileges: *ref_0
- names:
- metrics-system.diskio-test_ns
privileges: *ref_0
- names:
- metrics-system.socket_summary-test_ns
privileges: *ref_0
- names:
- metrics-system.process-test_ns
privileges: *ref_0
- names:
- metrics-system.load-test_ns
privileges: *ref_0
agent:
logging:
level: debug
monitoring:
enabled: true
use_output: default
namespace: test_ns
logs: true
metrics: true
providers:
kubernetes:
node: ${NODE_NAME}
scope: node
kubernetes_leaderelection:
enabled: false
inputs:
- name: apm-2
revision: 2
type: apm
use_output: default
meta:
package:
name: apm
version: 7.17.0
data_stream:
namespace: test_ns
apm-server:
capture_personal_data: true
max_connections: 0
max_event_size: 307200
auth:
api_key:
enabled: false
limit: 100
anonymous:
enabled: true
allow_agent:
- rum-js
- js-base
- iOS/swift
allow_service: null
rate_limit:
ip_limit: 1000
event_limit: 300
secret_token: null
default_service_environment: null
shutdown_timeout: 30s
sampling:
tail:
enabled: false
policies: null
rum:
enabled: true
exclude_from_grouping: ^/webpack
allow_headers: null
response_headers: null
library_pattern: node_modules|bower_components|~
allow_origins:
- '*'
source_mapping:
metadata: []
ssl:
enabled: false
key_passphrase: null
certificate: null
supported_protocols:
- TLSv1.0
- TLSv1.1
- TLSv1.2
curve_types: null
key: null
cipher_suites: null
response_headers: null
write_timeout: 30s
host: "0.0.0.0:8200"
max_header_size: 1048576
idle_timeout: 45s
expvar.enabled: false
read_timeout: 3600s
java_attacher:
enabled: null
discovery-rules: null
download-agent-version: null
agent_config:
- service: {}
etag: b52d6aa9cbcb5508c6edfb57cf7770090856f77f
config:
transaction_max_spans: '500'
transaction_sample_rate: '1.0'
- name: system-metrics
type: system/metrics
use_output: default
meta:
package:
name: system
version: 0.11.0
data_stream:
namespace: test_ns
streams:
- data_stream:
dataset: system.core
type: metrics
metricsets:
- core
core.metrics:
- percentages
- data_stream:
dataset: system.cpu
type: metrics
period: 10s
cpu.metrics:
- percentages
- normalized_percentages
metricsets:
- cpu
- data_stream:
dataset: system.diskio
type: metrics
period: 10s
diskio.include_devices: null
metricsets:
- diskio
- data_stream:
dataset: system.filesystem
type: metrics
period: 1m
metricsets:
- filesystem
processors:
- drop_event.when.regexp:
system.filesystem.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
- data_stream:
dataset: system.fsstat
type: metrics
period: 1m
metricsets:
- fsstat
processors:
- drop_event.when.regexp:
system.fsstat.mount_point: ^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)
- data_stream:
dataset: system.load
type: metrics
period: 10s
metricsets:
- load
- data_stream:
dataset: system.memory
type: metrics
period: 10s
metricsets:
- memory
- data_stream:
dataset: system.network
type: metrics
period: 10s
network.interfaces: null
metricsets:
- network
- data_stream:
dataset: system.process
type: metrics
process.include_top_n.by_memory: 5
period: 10s
processes:
- .*
process.include_top_n.by_cpu: 5
process.cgroups.enabled: false
process.cmdline.cache.enabled: true
metricsets:
- process
process.include_cpu_ticks: false
system.hostfs: /hostfs
- data_stream:
dataset: system.process_summary
type: metrics
period: 10s
metricsets:
- process_summary
system.hostfs: /hostfs
- data_stream:
dataset: system.socket_summary
type: metrics
period: 10s
metricsets:
- socket_summary
system.hostfs: /hostfs
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: elastic-agent
namespace: kube-elasticagent
labels:
app: elastic-agent
spec:
selector:
matchLabels:
app: elastic-agent
template:
metadata:
labels:
app: elastic-agent
spec:
serviceAccountName: elastic-agent
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: elastic-agent
image: docker.elastic.co/beats/elastic-agent:7.17.6
args: [
"-c", "/etc/agent.yml",
"-e",
]
env:
- name: ES_USERNAME
value: "elastic"
- name: ES_PASSWORD
value: "changeme"
- name: ES_HOST
value: "elasticurl"
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: NODE_IP
valueFrom:
fieldRef:
fieldPath: status.hostIP
securityContext:
runAsUser: 0
ports:
- name: apm
containerPort: 8200
hostPort: 8200
resources:
limits:
memory: 500Mi
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: datastreams
mountPath: /etc/agent.yml
readOnly: true
subPath: agent.yml
- name: proc
mountPath: /hostfs/proc
readOnly: true
- name: cgroup
mountPath: /hostfs/sys/fs/cgroup
readOnly: true
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: varlog
mountPath: /var/log
readOnly: true
volumes:
- name: datastreams
configMap:
defaultMode: 0640
name: agent-node-datastreams
- name: proc
hostPath:
path: /proc
- name: cgroup
hostPath:
path: /sys/fs/cgroup
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: varlog
hostPath:
path: /var/log
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: elastic-agent
subjects:
- kind: ServiceAccount
name: elastic-agent
namespace: kube-elasticagent
roleRef:
kind: ClusterRole
name: elastic-agent
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
namespace: kube-elasticagent
name: elastic-agent
subjects:
- kind: ServiceAccount
name: elastic-agent
namespace: kube-elasticagent
roleRef:
kind: Role
name: elastic-agent
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: elastic-agent
labels:
k8s-app: elastic-agent
rules:
- apiGroups: [""]
resources:
- nodes
- namespaces
- events
- pods
- services
- configmaps
verbs: ["get", "list", "watch"]
- apiGroups: ["extensions"]
resources:
- replicasets
verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
resources:
- statefulsets
- deployments
- replicasets
verbs: ["get", "list", "watch"]
- apiGroups: ["batch"]
resources:
- jobs
verbs: ["get", "list", "watch"]
- apiGroups:
- ""
resources:
- nodes/stats
verbs:
- get
# required for apiserver
- nonResourceURLs:
- "/metrics"
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: elastic-agent
# should be the namespace where elastic-agent is running
namespace: kube-elasticagent
labels:
k8s-app: elastic-agent
rules:
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs: ["get", "create", "update"]
- apiGroups: ['policy']
resources: ['podsecuritypolicies']
verbs: ['use']
resourceNames:
- elastic-agent
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: elastic-agent
namespace: kube-elasticagent
labels:
k8s-app: elastic-agent
---
apiVersion: policy/v1beta1
kind: PodSecurityPolicy
metadata:
name: elastic-agent
spec:
allowPrivilegeEscalation: false
allowedHostPaths:
- pathPrefix: /var/lib/docker/containers
readOnly: true
- pathPrefix: /var/log
readOnly: true
- pathPrefix: /proc
readOnly: true
- pathPrefix: /sys/fs/cgroup
readOnly: true
fsGroup:
rule: RunAsAny
runAsUser:
rule: RunAsAny
seLinux:
rule: RunAsAny
supplementalGroups:
rule: RunAsAny
volumes:
- configMap
- secret
- hostPath
- emptyDir
hostNetwork: true
hostPorts:
- min: 8200
max: 8200
I got same behaviour on docker, so here are my docker files, in case is easier to test.
docker-compose.yml
version: "3"
services:
elastic-agent:
image: docker.elastic.co/beats/elastic-agent:7.17.7
container_name: elastic-agent
restart: always
user: root # note, synthetic browser monitors require this set to `elastic-agent`
volumes:
- ./elastic-agent.yml:/usr/share/elastic-agent/elastic-agent.yml
ports:
- 8200:8200
elastic-agent.yml
###################### Agent Configuration Example #########################
# This file is an example configuration file highlighting only the most common
# options. The elastic-agent.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
######################################
# Fleet configuration
######################################
outputs:
default:
type: elasticsearch
hosts: [elasticurl]
username: elastic
password: changeme
ssl.verification_mode: none
inputs:
- type: system/metrics
# Namespace name must conform to the naming conventions for Elasticsearch indices, cannot contain dashes (-), and cannot exceed 100 bytes
# For index naming restrictions, see https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#indices-create-api-path-params
data_stream.namespace: local_test
use_output: default
streams:
- metricset: cpu
# Dataset name must conform to the naming conventions for Elasticsearch indices, cannot contain dashes (-), and cannot exceed 100 bytes
data_stream.dataset: system.cpu
- metricset: memory
data_stream.dataset: system.memory
- metricset: network
data_stream.dataset: system.network
- metricset: filesystem
data_stream.dataset: system.filesystem
- name: apm-2
revision: 2
type: apm
use_output: default
meta:
package:
name: apm
version: 7.17.0
data_stream:
namespace: local_test
apm-server:
capture_personal_data: true
max_connections: 0
max_event_size: 307200
auth:
api_key:
enabled: true
limit: 100
anonymous:
enabled: true
allow_agent:
- rum-js
- js-base
- iOS/swift
allow_service: null
rate_limit:
ip_limit: 1000
event_limit: 300
secret_token: abcd1234
default_service_environment: null
shutdown_timeout: 30s
sampling:
tail:
enabled: false
policies: null
rum:
enabled: true
exclude_from_grouping: ^/webpack
allow_headers: null
response_headers: null
library_pattern: node_modules|bower_components|~
allow_origins:
- '*'
source_mapping:
metadata: []
ssl:
enabled: false
key_passphrase: null
certificate: null
supported_protocols:
- TLSv1.0
- TLSv1.1
- TLSv1.2
curve_types: null
key: null
cipher_suites: null
response_headers: null
write_timeout: 30s
host: "0.0.0.0:8200"
max_header_size: 1048576
idle_timeout: 45s
expvar.enabled: false
read_timeout: 3600s
java_attacher:
enabled: null
discovery-rules: null
download-agent-version: null
agent_config:
- service: {}
config:
transaction_max_spans: '500'
transaction_sample_rate: '1.0'
# Send all logging output to stderr. The default is false.
agent.logging.to_stderr: true
Elastic url, username, and password are set up correctly. I get system metrics on my elastic so metricbeats does work fine.
Let me know if i can provide any more info.
axw
November 2, 2022, 6:44am
4
@juliana_cg I can reproduce the symptoms you're seeing. I haven't gotten to the bottom of why this is happening; there's likely some assumptions built in about Fleet.
In case you missed it, APM Server doesn't currently support Elastic Agent standalone, as mentioned in a note at Quick start | APM User Guide [8.5] | Elastic
We are thinking about making some changes to enable that. In the mean time, is running Legacy APM Server an option for you? Getting started with APM Server | APM User Guide [8.5] | Elastic
Hey @axw
In case you missed it, APM Server doesn't currently support Elastic Agent standalone, as mentioned in a note at Quick start | APM User Guide [8.5] | Elastic
Totally missed this
In the mean time, is running Legacy APM Server an option for you?
I'm not sure, i get this on APM's app settings
So I assumed I couldn't use legacy apm anymore, is this correct?
axw
November 3, 2022, 2:03am
6
I'm not sure, i get this on APM's app settings
Ah, I see. If you have migrated to data streams, then you should really be setting up Elastic Agent with Fleet. However...
So I assumed I couldn't use legacy apm anymore, is this correct?
In 7.17 you can configure legacy APM Server to write to data streams by setting apm-server.data_streams.enabled: true in apm-server.yml. This is the default in 8.0+
1 Like
Oh, i see. That's great news. Will try to deploy apm server on kubernetes then. Thanks a lot!
system
December 1, 2022, 9:25am
8
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
it may not be present on 7.17 docs