Thousands of predefined fields visible in "Default fields" in discover

Elastic Stack Kibana
1

I'm running Kibana/Elasticsearch/FileBeat 8.12.0 on Kubernetes, deployed using ECK operator.
I'm only collecting logs from selected workloads in the kubernetes cluster.
My filebeat configuration looks like this:

    filebeat:
      modules: []
      overwrite_pipelines: true
      autodiscover:
        providers:
        - type: kubernetes
          node: ${NODE_NAME}
          add_resource_metadata:
            node:
              enabled: false
            cronjob: true
            deployment: true
          hints:
            enabled: true
            default_config:
              enabled: false
              type: container
              paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log

My index contains less than 50 fields.
However in the Discover view of Kibana, "Available fields" sections shows 7223 entries. Name prefixes of the fields look like they could be related to Filebetat plugins: activemq, aws, awscloudwatch, azure and so on.

I'm looking for a way of configuring FIlebeat and/or Kibana to show only the fields that actually exists on my index.

My attempts to disable Filebeat plugins didn't help. I can see many entries in the Mappings pane in Stack Management > Index Management > Templates > filebeat-8.12.0 > Edit screen and I guess I could try deleting them, but I think my changes will get reverted by the ECK operator.

Any pointers how to sort that out appreciated!

2

Hi @rkrzewski,

Welcome to the community!

Sorry for the inconvenience! This was improved in 8.13 [UnifiedFieldList] Categorize fields as empty that never had a value in matching indices by kertal · Pull Request #174063 · elastic/kibana · GitHub

As a workaround for 8.12, Data View Field filters could be defined on Stack Management > Data Views > Your data view > Field filters tab.

3

Thanks so much Julia!

I've upgraded my ELK stack to 8.13.0 using ECK operator and it did resolve the issue.

1 Like