Unable to see the "Available fields column" in the logs forwarding for Devbyok cluster

[subagh]

Pic 1 - Does not shows any available fields in Dashboard

Can anyone please tell me why I cannot see the relevant "string fields" tab to select options from DevByok cluster but with similar configuration, I am able to see these in Test Byok.

[warkolm (Mark Walkom)]

Welcome to our community!

Do the mappings match across both indices?

[subagh]

yes they matches the same , indices pattern , cluster which has similar Elasticsearch and fluentbit configuration.
What i wish to know is that - is there any field in fluenbit config file or Elasticsearch yaml file that is responsible for display of "Available Fields"?
Fluentbit config file - DevBYOK

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
  labels:
    k8s-app: fluent-bit
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
        Flush         1
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    @INCLUDE input-kubernetes.conf
    @INCLUDE filter-kubernetes.conf
    @INCLUDE output-elasticsearch.conf

  input-kubernetes.conf: |
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Exclude_Path      /var/log/containers/tiller*.log,/var/log/containers/kube*.log,/var/log/containers/coredns-*.log,/var/log/containers/azure-cni-networkmonitor*.log,/var/log/containers/fluent-bit*.log,/var/log/containers/elastic*.log,/var/log/containers/istio*.log,/var/log/containers/metrics-server*.log,/var/log/containers/prometheus*.log,/var/log/containers/alertmanager*.log,/var/log/containers/node-exporter*.log,/var/log/containers/prometheus-operator*.log,/var/log/containers/prometheus-config-reloader*.log,/var/log/containers/grafana-watcher*.log,/var/log/containers/metrics-scraper*.log,/var/log/containers/kube-proxy*.log,/var/log/containers/ip-masq-agent*.log,/var/log/containers/coredns*.log,/var/log/containers/networkmonitor*.log,/var/log/containers/kube-state-metrics*.log,/var/log/containers/grafana*.log
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     1000MB
        Skip_Long_Lines   On
        Refresh_Interval  10
        Buffer_Chunk_Size 64KB
        Buffer_Max_Size   128KB

  filter-kubernetes.conf: |
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off
        Buffer_Size         64KB

  output-elasticsearch.conf: |
    [OUTPUT]
        Name            es
        Match           kube.*
        Host            10.0.72.0
        Port            9200
        HTTP_User       elastic
        HTTP_Passwd     ************
        Index           devbyok
        Logstash_Format On
        Logstash_Prefix devbyok
        Replace_Dots    Off
        Retry_Limit     5
        tls             On
        tls.verify      Off
        Trace_Error     On


  parsers.conf: |
    [PARSER]
        Name        docker
        Format      json
        Time_Key    time
        Time_Format %Y-%m-%dT%H:%M:%S.%L
        Time_Keep   On

[warkolm (Mark Walkom)]

Kibana will show all fields that it finds in the Elasticsearch index.