Hello,
I have configured Fluentbit in my k8s cluster to send logs to S3 and ELK stack to get logs from S3 and Visualise in Kibana. When I create Dataview in kibana for the Index, the kibana dashboard is not giving filters related to Kubernetes (e.g. kubernetes.namespace_name).
Following is my configuration of Fluentbit :
config:
service: |
[SERVICE]
Daemon Off
Flush {{ .Values.flush }}
Log_Level {{ .Values.logLevel }}
Parsers_File parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port {{ .Values.metricsPort }}
Health_Check On
storage.sync normal
storage.checksum off
@INCLUDE application-log.conf
inputs: |
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
Parser docker
Mem_Buf_Limit 50MB
Skip_Long_Lines On
Refresh_Interval 10
filters: |
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Merge_Log On
Merge_Log_Trim On
K8S-Logging.Parser On
K8S-Logging.Exclude On
Annotations Off
Labels On
[FILTER]
Name modify
Match *
Remove kubernetes.container_hash
Remove kubernetes.docker_id
Remove kubernetes.pod_id
outputs: |
[OUTPUT]
Name s3
Match *
bucket mys3
region us-east-2
use_put_object On
compression gzip
total_file_size 250M
upload_timeout 2m
s3_key_format /$TAG[2]/$TAG[3]/%Y/%m/%d/$TAG[4]__$TAG[5]/$UUID-%M-%S.gz
s3_key_format_tag_delimiters ._
customParsers: |
[PARSER]
Name docker
Format json
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
Time_Keep On
upstream: {}
Please suggest if any other config needs to done.
Thanks
