We are using hints based auto discovery for metricteat 7.13.0 on ECK 1.6.0, we keep geeting this error:
ERROR [autodiscover] autodiscover/autodiscover.go:212 Auto discover config check failed for config '{
"metricsets": [
"pending_tasks",
"node_stats"
],
"module": "elasticsearch",
"period": "1m",
"timeout": "3s"
}', won't start runner: 4 errors: host parsing failed for elasticsearch-shard: error parsing URL: empty host; host parsing failed for elasticsearch-node: error parsing URL: empty host; host parsing failed for elasticsearch-pending_tasks: error parsing URL: empty host; host parsing failed for elasticsearch-node_stats: error parsing URL: empty hostERROR [autodiscover] autodiscover/autodiscover.go:212 Auto discover config check failed for config '{
"enabled": true,
"metricsets": [
"node",
"pending_tasks",
"shard",
"node_stats"
],
We are using the following configurations and I also want to put into consideration that cluster monitoring is working for all components.
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
name: elastic-mb
namespace: elasticsearch
spec:
type: metricbeat
version: 7.13.0
elasticsearchRef:
name: elastic
kibanaRef:
name: elastic
config:
logging.json: true
logging.level: warning
monitoring.enabled: true
metricbeat:
autodiscover:
providers:
- type: kubernetes
scope: cluster
hints.enabled: true
templates:
- condition:
contains:
kubernetes.labels.elastic-name: elastic
config:
- module: elasticsearch
metricsets:
- ccr
- cluster_stats
- enrich
- index
- index_recovery
- index_summary
- ml_job
- node_stats
- shard
period: 10s
hosts: "https://${data.host}:${data.ports.https}"
username: ${MONITORED_ES_USERNAME}
password: ${MONITORED_ES_PASSWORD}
ssl.certificate_authorities:
- /mnt/elastic-internal/elasticsearch-certs/ca.crt
xpack.enabled: true
- condition:
contains:
kubernetes.labels.kibana-name: elastic
config:
- module: kibana
metricsets:
- stats
period: 10s
hosts: "https://${data.host}:${data.ports.https}"
username: ${MONITORED_ES_USERNAME}
password: ${MONITORED_ES_PASSWORD}
ssl.certificate_authorities:
- /mnt/elastic-internal/kibana-certs/ca.crt
xpack.enabled: true
- type: kubernetes
scope: cluster
node: ${NODE_NAME}
unique: true
templates:
- config:
- module: kubernetes
hosts: [ "kube-state-metrics.kube-system:8080" ]
period: 10s
add_metadata: true
metricsets:
- state_node
- state_deployment
- state_daemonset
- state_replicaset
- state_pod
- state_container
- state_cronjob
- state_resourcequota
- state_statefulset
- state_service
- module: kubernetes
metricsets:
- apiserver
hosts: [ "https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}" ]
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
ssl.certificate_authorities:
- /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
period: 30s
modules:
- module: system
period: 10s
metricsets:
- cpu
- load
- memory
- network
- process
- process_summary
processes: [ '.*' ]
process.include_top_n:
by_cpu: 5 # include top 5 processes by CPU
by_memory: 5 # include top 5 processes by memory
- module: system
period: 1m
metricsets:
- filesystem
- fsstat
processors:
- drop_event.when.regexp:
system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib|snap)($|/)'
- module: kubernetes
metricsets:
- node
- system
- pod
- container
- volume
period: 10s
host: ${NODE_NAME}
hosts: [ "https://${NODE_NAME}:10250" ]
bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
ssl.certificate_authorities:
- /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
- module: kubernetes
metricsets:
- proxy
period: 10s
host: ${NODE_NAME}
hosts: [ "localhost:10249" ]
processors:
- add_cloud_metadata: {}
- add_host_metadata: {}
daemonSet:
podTemplate:
spec:
serviceAccountName: metricbeat
automountServiceAccountToken: true
terminationGracePeriodSeconds: 30
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: metricbeat
resources:
limits:
memory: 500Mi
requests:
cpu: 100m
memory: 200Mi
args: [
"-c", "/etc/beat.yml",
"-e",
"-system.hostfs=/hostfs",
]
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: MONITORED_ES_USERNAME
value: elastic
- name: MONITORED_ES_PASSWORD
valueFrom:
secretKeyRef:
key: elastic
name: elastic-es-elastic-user
securityContext:
runAsUser: 0
volumeMounts:
- name: data
mountPath: /usr/share/metricbeat/data
- name: proc
mountPath: /hostfs/proc
readOnly: true
- name: cgroup
mountPath: /hostfs/sys/fs/cgroup
readOnly: true
volumes:
- name: proc
hostPath:
path: /proc
- name: cgroup
hostPath:
path: /sys/fs/cgroup
- name: data
hostPath:
path: /var/lib/metricbeat-data
type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: metricbeat
rules:
- apiGroups: [""]
resources:
- nodes
- namespaces
- events
- pods
- secrets
- services
verbs: ["get", "list", "watch"]
- apiGroups: ["extensions"]
resources:
- replicasets
verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
resources:
- statefulsets
- deployments
- replicasets
verbs: ["get", "list", "watch"]
- apiGroups:
- ""
resources:
- nodes/stats
verbs:
- get
- apiGroups:
- coordination.k8s.io
resources:
- leases
verbs:
- '*'
- nonResourceURLs:
- "/metrics"
verbs:
- get
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: metricbeat
namespace: elasticsearch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: metricbeat
subjects:
- kind: ServiceAccount
name: metricbeat
namespace: elasticsearch
roleRef:
kind: ClusterRole
name: metricbeat
apiGroup: rbac.authorization.k8s.io
Please also note that the autodiscovery rule configured is working, and it is diffrent from what is reporting error, u can notice the period of 1m compared to 10s configured in the yaml and the metricsets also.
Can you help me to identify why this is happening and how to solve.
Thank you very much