rkelastic
(ElasticRK)
April 21, 2023, 6:31am
1
I am using a filebeat instance to read data from azure blob storage and see the visualisation in kibana.
In Discover tab in kibana, under the "Available fields" section, it is showing list of fields which are not there in the data present in blob storage.
I only have 6 fields in the data i am sending.
Can anyone help me with it
Tre_Seymour
(Wayne Kenneth Seymour III (Tre'))
April 21, 2023, 1:46pm
2
Hello there @rkelastic
Before we get started, may I ask which version you are using?
rkelastic
(ElasticRK)
April 23, 2023, 12:16pm
3
the versions are 8.7.0 (same for kibana, elasticsearch and filebeat)
i have to files for config:
filebeat.yml
ingestion_pipeline in Elasticsearch
Here is the filebeat yml file that i am using
filebeat.inputs:
- type: azure-blob-storage
id: my-azureblobstorage-id
enabled: true
account_name: <acc_name>
auth.connection_string.uri: <conn_String>
max_workers: 10
poll: true
poll_interval: 15s
containers:
- name: testlogstash
max_workers: 3
poll: true
poll_interval: 2s
processors:
- drop_fields:
fields: ["@timestamp", "input", "prospector", "host", "agent", "event", "ecs", "log"]
ignore_missing: true
- decode_json_fields:
fields: ["message"]
target: ""
overwrite_keys: true
output.elasticsearch:
hosts: '${ELASTICSEARCH_HOSTS:<ip>:9200}'
index: 'processed1-1'
pipeline: 'epoch-time-converter'
setup.template.name: "processed1"
setup.template.pattern: "processed1-1"
ingestion pipeline
{
"epoch-time-converter": {
"description": "test pipeline to parse timestamp field",
"processors": [
{
"date": {
"field": "timestamp",
"formats": [
"UNIX_MS"
],
"if": "ctx.timestamp != null && (ctx.timestamp instanceof String || ctx.timestamp instanceof Number)"
}
}
]
}
}
system
(system)
Closed
May 21, 2023, 12:16pm
4
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.