I am trying to setup filebeat with autodiscover, but container is failing with this error
2021-05-27T19:25:03.029Z ERROR instance/beat.go:971 Exiting: error in autodiscover provider settings: error setting up kubernetes autodiscover provider: missing field accessing 'filebeat.autodiscover.providers.0.host' (source:'/etc/filebeat.yml')
Exiting: error in autodiscover provider settings: error setting up kubernetes autodiscover provider: missing field accessing 'filebeat.autodiscover.providers.0.host' (source:'/etc/filebeat.yml')
I have following config
Config
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: kube-logging
labels:
app: filebeat
data:
filebeat.yml: |-
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
#reload.period: 10s
filebeat.autodiscover:
providers:
- type: kubernetes
host: ${NODE_NAME}
hints.enabled: true
setup.template.settings:
index.mapping.total_fields.limit: 5000
processors:
- add_cloud_metadata:
- add_host_metadata:
- add_docker_metadata:
- add_kubernetes_metadata:
- drop_event:
when:
or:
- contains:
message: "OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain"
- and:
- equals:
kubernetes.container.name: "metricbeat"
- contains:
message: "INFO"
- contains:
message: "Non-zero metrics in the last"
- and:
- equals:
kubernetes.container.name: "packetbeat"
- contains:
message: "INFO"
- contains:
message: "Non-zero metrics in the last"
- contains:
message: "get services heapster"
- contains:
kubernetes.container.name: "kube-addon-manager"
- contains:
kubernetes.container.name: "dashboard-metrics-scraper"
output.elasticsearch:
hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
logging.level: warning
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-inputs
namespace: kube-logging
labels:
app: filebeat
data:
kubernetes.yml: |-
- type: container
containers.ids:
- "*"
processors:
- add_kubernetes_metadata:
in_cluster: true
- add_id: ~
---
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-modules
namespace: kube-logging
labels:
app: filebeat
data:
elasticsearch.yml: |-
- module: elasticsearch
traefik.yml: |-
- module: traefik
access:
enabled: true
var.paths: ["/usr/local/traefik/access.log*"]
nginx.yml: |-
- module: nginx
access:
enabled: true
var.paths: ["/path/to/log/nginx/access.log*"]
error:
enabled: true
var.paths: ["/path/to/log/nginx/error.log*"]
DaemonSet
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: filebeat
namespace: kube-logging
labels:
app: filebeat
spec:
selector:
matchLabels:
app: filebeat
minReadySeconds: 10
updateStrategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
template:
metadata:
labels:
app: filebeat
spec:
serviceAccountName: filebeat
terminationGracePeriodSeconds: 30
containers:
- name: filebeat
image: docker.elastic.co/beats/filebeat:7.12.1
args: [
"-c", "/etc/filebeat.yml",
"-e",
]
env:
- name: ELASTICSEARCH_HOST
value: elasticsearch
- name: ELASTICSEARCH_PORT
value: "9200"
- name: ELASTICSEARCH_USERNAME
value: elastic
- name: ELASTICSEARCH_PASSWORD
value: changeme
securityContext:
runAsUser: 0
resources:
limits:
memory: 1800Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config
mountPath: /etc/filebeat.yml
readOnly: true
subPath: filebeat.yml
- name: inputs
mountPath: /usr/share/filebeat/inputs.d
readOnly: true
- name: data
mountPath: /usr/share/filebeat/data
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
volumes:
- name: config
configMap:
defaultMode: 0600
name: filebeat-config
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: inputs
configMap:
defaultMode: 0600
name: filebeat-inputs
# data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
- name: data
hostPath:
path: /var/lib/filebeat-data
type: DirectoryOrCreate
Version :
filebeat:7.12.1
what I am doing wrong here ?