Not Getting Kubernetes related filters in Kibana when exporting logs from Logstash

Elastic Stack Kibana
1

Hello,

I have configured Fluentbit in my k8s cluster to send logs to S3 and ELK stack to get logs from S3 and Visualise in Kibana. When I create Dataview in kibana for the Index, the kibana dashboard is not giving filters related to Kubernetes (e.g. kubernetes.namespace_name).

Following is my configuration of Fluentbit :

config:
  service: |
    [SERVICE]
        Daemon Off
        Flush {{ .Values.flush }}
        Log_Level {{ .Values.logLevel }}
        Parsers_File parsers.conf
        HTTP_Server On
        HTTP_Listen 0.0.0.0
        HTTP_Port {{ .Values.metricsPort }}
        Health_Check On
        storage.sync              normal
        storage.checksum          off
        @INCLUDE application-log.conf        

  inputs: |
    [INPUT]
        Name                tail
        Tag                 kube.*
        Path                /var/log/containers/*.log
        Parser              docker
        Mem_Buf_Limit       50MB
        Skip_Long_Lines     On
        Refresh_Interval    10        

  filters: |
    [FILTER]
        Name                  kubernetes
        Match                 kube.*
        Kube_URL              https://kubernetes.default.svc:443
        Merge_Log             On
        Merge_Log_Trim        On
        K8S-Logging.Parser    On
        K8S-Logging.Exclude   On
        Annotations           Off
        Labels                On

    [FILTER]
        Name    modify
        Match   *
        Remove  kubernetes.container_hash
        Remove  kubernetes.docker_id
        Remove  kubernetes.pod_id

  outputs: |
    [OUTPUT]
        Name                          s3
        Match                         *
        bucket                        mys3
        region                        us-east-2
        use_put_object                On
        compression                   gzip
        total_file_size               250M
        upload_timeout                2m
        s3_key_format                 /$TAG[2]/$TAG[3]/%Y/%m/%d/$TAG[4]__$TAG[5]/$UUID-%M-%S.gz
        s3_key_format_tag_delimiters  ._



  customParsers: |
    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

  upstream: {}

Please suggest if any other config needs to done.

Thanks

2

Welcome to our community! :smiley:

What does one of the events look like when you view it in Discover?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.