Not Getting Kubernetes related filters in Kibana when exporting logs from Logstash


I have configured Fluentbit in my k8s cluster to send logs to S3 and ELK stack to get logs from S3 and Visualise in Kibana. When I create Dataview in kibana for the Index, the kibana dashboard is not giving filters related to Kubernetes (e.g. kubernetes.namespace_name).

Following is my configuration of Fluentbit :

  service: |
        Daemon Off
        Flush {{ .Values.flush }}
        Log_Level {{ .Values.logLevel }}
        Parsers_File parsers.conf
        HTTP_Server On
        HTTP_Port {{ .Values.metricsPort }}
        Health_Check On
        storage.sync              normal
        storage.checksum          off
        @INCLUDE application-log.conf        

  inputs: |
        Name                tail
        Tag                 kube.*
        Path                /var/log/containers/*.log
        Parser              docker
        Mem_Buf_Limit       50MB
        Skip_Long_Lines     On
        Refresh_Interval    10        

  filters: |
        Name                  kubernetes
        Match                 kube.*
        Kube_URL              https://kubernetes.default.svc:443
        Merge_Log             On
        Merge_Log_Trim        On
        K8S-Logging.Parser    On
        K8S-Logging.Exclude   On
        Annotations           Off
        Labels                On

        Name    modify
        Match   *
        Remove  kubernetes.container_hash
        Remove  kubernetes.docker_id
        Remove  kubernetes.pod_id

  outputs: |
        Name                          s3
        Match                         *
        bucket                        mys3
        region                        us-east-2
        use_put_object                On
        compression                   gzip
        total_file_size               250M
        upload_timeout                2m
        s3_key_format                 /$TAG[2]/$TAG[3]/%Y/%m/%d/$TAG[4]__$TAG[5]/$UUID-%M-%S.gz
        s3_key_format_tag_delimiters  ._

  customParsers: |
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

  upstream: {}

Please suggest if any other config needs to done.


Welcome to our community! :smiley:

What does one of the events look like when you view it in Discover?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.