Configure output Index name with kubernetes metadata field

Elastic Stack Beats
1

Hi,

I would like to have the kubernetes namespace in the index name, so I tried like this :

output.elasticsearch:
  hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']  
  index: "%{[kubernetes.namespace]}-filebeat-%{[beat.version]}-%{+yyyy.MM.dd}"

With this configuration, no index at all is created even if the metadata are present in the documents :

018-06-19T21:30:39.424Z	DEBUG	[publish]	pipeline/processor.go:275	Publish event: {
  "@timestamp": "2018-06-19T21:30:38.483Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "doc",
    "version": "6.2.4"
  },
  "message": "      }",
  "source": "/var/lib/docker/containers/af869caadf53af083955fee5e506fd2ab6658ea7e285c12a8c724462b347494c/af869caadf53af083955fee5e506fd2ab6658ea7e285c12a8c724462b347494c-json.log",
  "offset": 535365,
  "prospector": {
    "type": "docker"
  },
  "docker": {
    "container": {
      "id": "af869caadf53af083955fee5e506fd2ab6658ea7e285c12a8c724462b347494c",
      "name": "k8s_filebeat_filebeat-958s5_default_01d6839d-7408-11e8-9939-42010a8e0061_0",
      "image": "docker.elastic.co/beats/filebeat@sha256:1ab075d016257ee7990f962a11099284e58b00f3d474f042c5d174fd48809eaa",
      "labels": {
        "io.kubernetes.pod.name": "filebeat-958s5",
        "org.label-schema.schema-version": "= 1.0     org.label-schema.name=CentOS Base Image     org.label-schema.vendor=CentOS     org.label-schema.license=GPLv2     org.label-schema.build-date=20180402",
        "io.kubernetes.sandbox.id": "2237901ac21fcc16fb9bc2b69ccbb43dfe3b22916860bd0267da007c649ae478",
        "annotation.io.kubernetes.container.terminationMessagePath": "/dev/termination-log",
        "annotation.io.kubernetes.container.terminationMessagePolicy": "File",
        "annotation.io.kubernetes.pod.terminationGracePeriod": "30",
        "io.kubernetes.container.name": "filebeat",
        "io.kubernetes.container.logpath": "/var/log/pods/01d6839d-7408-11e8-9939-42010a8e0061/filebeat_0.log",
        "io.kubernetes.pod.namespace": "default",
        "io.kubernetes.pod.uid": "01d6839d-7408-11e8-9939-42010a8e0061",
        "annotation.io.kubernetes.container.hash": "629c5c7d",
        "io.kubernetes.docker.type": "container",
        "annotation.io.kubernetes.container.restartCount": "0"
      }
    }
  },
  "kubernetes": {
    "pod": {
      "name": "filebeat-958s5"
    },
    "node": {
      "name": "gke-cluster-1-default-pool-8f121b79-qrxv"
    },
    "namespace": "default",
    "labels": {
      "pod-template-generation": "2",
      "app": "filebeat",
      "controller-revision-hash": "3319485886"
    },
    "container": {
      "name": "filebeat"
    }
  },
  "beat": {
    "name": "filebeat-958s5",
    "hostname": "filebeat-958s5",
    "version": "6.2.4"
  },
  "stream": "stderr"
}
2

Hi @orgoz,

Your config looks good to me. Did you enable automatic index creation in Elasticsearch? Do you get any errors? Log output would be interesting here.

Best regards

3

Hello @exekias,

After trying two more times to send you logs, I just manage to get it running... I don't why, I have probably fix an error in my config file...

Sorry for the inconvenience

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.