Random fields in "Available fields" in Kibana discover

I am using a filebeat instance to read data from azure blob storage and see the visualisation in kibana.

In Discover tab in kibana, under the "Available fields" section, it is showing list of fields which are not there in the data present in blob storage.

I only have 6 fields in the data i am sending.

Can anyone help me with it

Hello there @rkelastic
Before we get started, may I ask which version you are using?

the versions are 8.7.0 (same for kibana, elasticsearch and filebeat)

i have to files for config:
filebeat.yml
ingestion_pipeline in Elasticsearch

Here is the filebeat yml file that i am using

filebeat.inputs:
- type: azure-blob-storage
  id: my-azureblobstorage-id
  enabled: true
  account_name: <acc_name>
  auth.connection_string.uri: <conn_String>
  max_workers: 10
  poll: true
  poll_interval: 15s
  containers:
  - name: testlogstash
    max_workers: 3
    poll: true
    poll_interval: 2s

processors:
- drop_fields:
    fields: ["@timestamp", "input", "prospector", "host", "agent", "event", "ecs", "log"]
    ignore_missing: true
- decode_json_fields:
    fields: ["message"]
    target: ""
    overwrite_keys: true


output.elasticsearch:
  hosts: '${ELASTICSEARCH_HOSTS:<ip>:9200}'
  index: 'processed1-1'
  pipeline: 'epoch-time-converter'

setup.template.name: "processed1"
setup.template.pattern: "processed1-1"

ingestion pipeline

{
  "epoch-time-converter": {
    "description": "test pipeline to parse timestamp field",
    "processors": [
      {
        "date": {
          "field": "timestamp",
          "formats": [
            "UNIX_MS"
          ],
          "if": "ctx.timestamp != null && (ctx.timestamp instanceof String || ctx.timestamp instanceof Number)"
        }
      }
    ]
  }
}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.