Hi
I am using filebeat and metric beat 7.2 and installing them on different windows and linux server. I want my beat agent to use existing ILM setting on elasticsearch but its overwriting it every time it is installed on new server.
Below is my configuration. Looks like this is not working.
setup.ilm.enabled: auto
# Configure the ILM write alias name.
setup.ilm.rollover_alias: "metricbeat"
# Configure rollover index pattern.
setup.ilm.pattern: "{now/d}-000001"
setup.ilm.check_exists: true
setup.ilm.overwrite: false
hey @syedsfayaz
so ILM policies are overwritten every time new beat is installed right?
is calling filebeat setup --ilm-policy part of your install script/procedure?
How does your beat ES output configuration looks like?
This is the command I am using part of install script for both windows and linux. I am seeing same issue with metric,filebeat.
filebeat setup -e
.\filebeat.exe setup
Can you tell me what exactly you are looking for when you say ES output configuration.
each beat is configured with an output such as output.elasticsearch
I was curious to see this section (redacted of sensitive data)
output.elasticsearch:
# Array of hosts to connect to.
hosts: ["monitor.example.com:9200"]
# Optional protocol and basic auth credentials.
#protocol: "https"
username: "elastic"
password: "*******"
Can you skim the beat logs for ILM mentions.
I assume you're not running OSS version of ES and ILM is available and enabled.
I see this in logs which should not be done as we have mentioned in the config not to overwrite if a template exists. Existing template will be overwritten, as overwrite is enabled.
I did not get this "I assume you're not running OSS version of ES and ILM is available and enabled." but as per my understanding the ES I am running is from a tar file.
`
2019-07-16T06:31:02.736-0700 INFO instance/beat.go:292 Setup Beat: metricbeat; Version: 7.2.0
2019-07-16T06:31:02.736-0700 INFO [index-management] idxmgmt/std.go:178 Set output.elasticsearch.index to 'metricbeat' as ILM is enabled.
2019-07-16T06:31:02.737-0700 INFO elasticsearch/client.go:166 Elasticsearch url: http://monitor.example.com:9200
2019-07-16T06:31:02.737-0700 INFO [publisher] pipeline/module.go:97 Beat name: rp.example.com
2019-07-16T06:31:02.737-0700 INFO elasticsearch/client.go:166 Elasticsearch url: http://monitor.example.com:9200
2019-07-16T06:31:02.743-0700 INFO elasticsearch/client.go:735 Attempting to connect to Elasticsearch version 7.2.0
2019-07-16T06:31:02.849-0700 INFO [index-management] idxmgmt/std.go:252 Auto ILM enable success.
2019-07-16T06:31:02.893-0700 INFO [index-management] idxmgmt/std.go:265 ILM policy successfully loaded.
2019-07-16T06:31:02.893-0700 INFO [index-management] idxmgmt/std.go:394 Set setup.template.name to '{metricbeat {now/d}-000001}' as ILM is enabled.
2019-07-16T06:31:02.893-0700 INFO [index-management] idxmgmt/std.go:399 Set setup.template.pattern to 'metricbeat-*' as ILM is enabled.
2019-07-16T06:31:02.893-0700 INFO [index-management] idxmgmt/std.go:433 Set settings.index.lifecycle.rollover_alias in template to {metricbeat {now/d}-000001} as ILM is enabled.
2019-07-16T06:31:02.893-0700 INFO [index-management] idxmgmt/std.go:437 Set settings.index.lifecycle.name in template to {metricbeat-7.2.0 {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2019-07-16T06:31:02.899-0700 INFO template/load.go:169 Existing template will be overwritten, as overwrite is enabled.
2019-07-16T06:31:03.065-0700 INFO template/load.go:108 Try loading template metricbeat to Elasticsearch
2019-07-16T06:31:03.146-0700 INFO template/load.go:100 template with name 'metricbeat' loaded.
2019-07-16T06:31:03.146-0700 INFO [index-management] idxmgmt/std.go:289 Loaded index template.
2019-07-16T06:31:03.147-0700 INFO [index-management] idxmgmt/std.go:300 Write alias successfully generated.
Index setup finished.
@Michal_Pristas Hi Michal any idea why its not working?