Filebeat.autodiscover seems to cause terraform/kubernetes based deploy issues

cgutshall · January 10, 2020, 9:02pm

I wrote up a yml based deploy to handle added a filebeat pod to each node in our EKS Cluster. When deploying the following config the pods are up and running with logs getting sent to elasticsearch.

gist.github.com

https://gist.github.com/cgutshall/7c233cc7f9e2ab560346545f6df3be1f

kube_filebeat.yml

apiVersion: v1
    kind: ConfigMap
    metadata:
      name: filebeat-config
      namespace: kube-system
      labels:
        k8s-app: filebeat
    data:
      filebeat.yml: |-
        filebeat.autodiscover:

This file has been truncated. show original

However I went to re-implement this with Terraform, for which I have the following two files. One to deploy the configmap.

gist.github.com

https://gist.github.com/cgutshall/fa44bfd98edc3e68a6669285dcad288c

main.tf

provider "kubernetes" {}
resource "kubernetes_config_map" "filebeat-config" {
  metadata {
    name      = "filebeat-config"
    namespace = "kube-system"
    labels = {
      k8s-apps = "filebeat"
    }
  }

This file has been truncated. show original

And a the contents of the config file is pointing to.

gist.github.com

https://gist.github.com/cgutshall/ea322d018e481e8bf54f704189cdb23a

filebeat.yml

filebeat.autodiscover:
  providers:
    - type: kubernetes
      host: ${NODE_NAME}
      hints.enabled: true
      hints.default_config:
        type: container
        paths:
          - /var/log/containers/*${data.kubernetes.container.id}.log

This file has been truncated. show original

Using the terraform deploy causes the pods to crash with the following error:

2020-01-10T18:20:49.260Z	ERROR	instance/beat.go:916	Exiting: error in autodiscover provider settings: error setting up kubernetes autodiscover provider: unable to build kube config due to error: invalid configuration: no configuration has been provided
Exiting: error in autodiscover provider settings: error setting up kubernetes autodiscover provider: unable to build kube config due to error: invalid configuration: no configuration has been provided

Are there any thoughts as to why this would occur? It only seems to be an issue when using the filebeat.autodiscover module.

Thanks

system · February 7, 2020, 9:02pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Autodiscover issue in kubernetes Beats filebeat	7	3482	July 7, 2020
Kubernetes - ConfigMap Beats filebeat	10	4396	April 13, 2018
Problem to send data from filebeat autodiscover container to elastic search on kubernetes Beats filebeat	2	851	September 10, 2020
Filebeat autodiscovery: appenders Beats filebeat	7	1057	April 28, 2018
Issue with Filebeat Kubernetes Configuration for Ingress Logs Beats docker , filebeat	2	255	October 9, 2023

Filebeat.autodiscover seems to cause terraform/kubernetes based deploy issues

Related topics