Hi!
I have filebeat with multiple indexes and one particular index is causing the issue and other indexes are sending logs without delayi.e., Sending logs to Kibana is getting delayed by 6 hours for the particular index.
`filebeat.inputs:
- type: log
enabled: true
paths:
- /mnt/test*.log
json.message_key: msg
json.keys_under_root: true
json.ignore_decoding_error: true
document_type: api
fields:
type: test-api
filebeat.inputs:
- type: log
enabled: true
paths:
- /mnt/test1*.log
json.message_key: msg
json.keys_under_root: true
json.ignore_decoding_error: true
document_type: api
fields:
type: test1-api
output.elasticsearch:
bulk_max_size: 7000
workers: 10
protocol: "https"
template.name: filebeat
template.path: filebeat.template.json
index: "ecs-%{[fields.type]:other}-{+yyyy.MM.dd}"
setup.template.name: filebeat
setup.template.pattern: "filebeat-*"
setup.ilm.enabled: false
setup.ilm.check_exists: false
setup.template.settings:
index.number_of_shards: 1
index.number_of_replicas: 1
index.refresh_interval: "30s"
`
test1-api is generating large sized logs whereas test-api is relatively smaller. any suggestions to resolve the issue