Hello All,
I've a requirement wherein I would like to have single filebeat.yml and this will have different log paths and will direct the data to respective diffrent index according to path.
Now this filebeat.yml would be installed in various servers with installed filebeat service and then
all the log path that are same in all servers should always write data to current rollover index.
ex:path 1 logs->direct to->mis-log index
ex:path 2 logs->direct to->mis-monitoring-usecase index
etc
In my index template already I've defined the ILM and rollover policy.
TEMPLATE:
PUT _index_template/mis-monitoring-usecases
{
"template": {
"settings": {
"index": {
"lifecycle": {
"name": "mis-monitoring-common-policy",
"rollover_alias": "mis-monitoring-usecases"
},
"default_pipeline": "mis-usecases-ingest-pipeline",
"number_of_shards": "1",
"number_of_replicas": "0"
}
},
"mappings": {
"properties": {
}
}
},
"index_patterns": [
"mis-monitoring-usecases-*"
],
"composed_of": []
}
mis-monitoring-common-
Current filebeat.yml:(for two log paths direct to two seperate index,but dont follow rollover policy/pattern)
Current filebeat.yml:
filebeat.inputs:
- type: log
enabled: true
paths:
- /k/cpp/LOG_new/CMS/**/*.log
fields:
index: mis-monitoring-usecases
ignore_older: 1h
include_lines: ['UsecaseMonitoring.*\|']
multiline.type: pattern
multiline.pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}Z
multiline.negate: true
multiline.match: after
scan_frequency: 30s
harvester_limit: 100
close_inactive: 30m
close_removed: true
clean_removed: true
processors:
- add_host_metadata: null
- drop_fields:
when:
equals:
agent.type: filebeat
fields:
- agent.hostname
- agent.id
- agent.type
- agent.ephemeral_id
- agent.version
- log.offset
- log.flags
- input.type
- ecs.version
- host
- type: log
enabled: true
paths:
- /L/app/LOG_LOOT/Demo/*.log
fields:
index: mis-log
ignore_older: 1h
include_lines:
- ^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}Z ?(.*)
multiline.type: pattern
multiline.pattern: ^[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2}.[0-9]{3}Z
multiline.negate: true
multiline.match: after
scan_frequency: 30s
harvester_limit: 100
close_inactive: 30m
close_removed: true
clean_removed: true
reload.enabled: false
processors:
- add_host_metadata: null
- drop_fields:
when:
equals:
agent.type: filebeat
fields:
- agent.hostname
- agent.id
- agent.type
- agent.ephemeral_id
- agent.version
- log.offset
- log.flags
- input.type
- ecs.version
- host.os
- host.id
- host.mac
- host.architecture
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
setup.ilm.enabled: true
setup.template.enabled: false
setup.ilm.pattern: '{now/d}-000001'
setup.kibana:
host: http://abc:5601
output.elasticsearch:
hosts:
- http://abc:9200
index: "%{[fields.index]}"
monitoring.enabled: true
monitoring.elasticsearch: null
CHALLENGE:
The ILM is not followed i.e ideally it should make two index of this pattern,which is not created:
mis-monitoring-usecases-2023.03.22-000001
mis-log-2023.03.22-000001
Read documentation its says:
When index lifecycle management (ILM) is enabled, the default index is "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}-%{index_num}" , for example, "filebeat-7.9.3-2023-03-09-000001" . Custom index settings are ignored when ILM is enabled. If you’re sending events to a cluster that supports index lifecycle management, see Index lifecycle management (ILM) to learn how to change the index name.
If I wanted to create my own index name, I can't use the ILM anymore? ,what are the alternatives ,sending filebaet to logstash is not an option for me.
elk 7.9.1 version
Kindly suggest
Thanx