ILM settings not working

Elastic Stack Beats
,
1

Hi

I am using filebeat and metric beat 7.2 and installing them on different windows and linux server. I want my beat agent to use existing ILM setting on elasticsearch but its overwriting it every time it is installed on new server.

Below is my configuration. Looks like this is not working.
setup.ilm.enabled: auto

# Configure the ILM write alias name.
setup.ilm.rollover_alias: "metricbeat"

# Configure rollover index pattern.
setup.ilm.pattern: "{now/d}-000001"
setup.ilm.check_exists: true
setup.ilm.overwrite: false
2

hey @syedsfayaz
so ILM policies are overwritten every time new beat is installed right?
is calling filebeat setup --ilm-policy part of your install script/procedure?

How does your beat ES output configuration looks like?

3

This is the command I am using part of install script for both windows and linux. I am seeing same issue with metric,filebeat.

filebeat setup -e
.\filebeat.exe setup

Can you tell me what exactly you are looking for when you say ES output configuration.

4

each beat is configured with an output such as output.elasticsearch
I was curious to see this section (redacted of sensitive data)

5 
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["monitor.example.com:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  username: "elastic"
  password: "*******"
6

Can you skim the beat logs for ILM mentions.
I assume you're not running OSS version of ES and ILM is available and enabled.

7

I see this in logs which should not be done as we have mentioned in the config not to overwrite if a template exists. Existing template will be overwritten, as overwrite is enabled.

I did not get this "I assume you're not running OSS version of ES and ILM is available and enabled." but as per my understanding the ES I am running is from a tar file.
`

2019-07-16T06:31:02.736-0700	INFO	instance/beat.go:292	Setup Beat: metricbeat; Version: 7.2.0
2019-07-16T06:31:02.736-0700	INFO	[index-management]	idxmgmt/std.go:178	Set output.elasticsearch.index to 'metricbeat' as ILM is enabled.
2019-07-16T06:31:02.737-0700	INFO	elasticsearch/client.go:166	Elasticsearch url: http://monitor.example.com:9200
2019-07-16T06:31:02.737-0700	INFO	[publisher]	pipeline/module.go:97	Beat name: rp.example.com
2019-07-16T06:31:02.737-0700	INFO	elasticsearch/client.go:166	Elasticsearch url: http://monitor.example.com:9200
2019-07-16T06:31:02.743-0700	INFO	elasticsearch/client.go:735	Attempting to connect to Elasticsearch version 7.2.0
2019-07-16T06:31:02.849-0700	INFO	[index-management]	idxmgmt/std.go:252	Auto ILM enable success.
2019-07-16T06:31:02.893-0700	INFO	[index-management]	idxmgmt/std.go:265	ILM policy successfully loaded.
2019-07-16T06:31:02.893-0700	INFO	[index-management]	idxmgmt/std.go:394	Set setup.template.name to '{metricbeat {now/d}-000001}' as ILM is enabled.
2019-07-16T06:31:02.893-0700	INFO	[index-management]	idxmgmt/std.go:399	Set setup.template.pattern to 'metricbeat-*' as ILM is enabled.
2019-07-16T06:31:02.893-0700	INFO	[index-management]	idxmgmt/std.go:433	Set settings.index.lifecycle.rollover_alias in template to {metricbeat {now/d}-000001} as ILM is enabled.
2019-07-16T06:31:02.893-0700	INFO	[index-management]	idxmgmt/std.go:437	Set settings.index.lifecycle.name in template to {metricbeat-7.2.0 {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2019-07-16T06:31:02.899-0700	INFO	template/load.go:169	Existing template will be overwritten, as overwrite is enabled.
2019-07-16T06:31:03.065-0700	INFO	template/load.go:108	Try loading template metricbeat to Elasticsearch
2019-07-16T06:31:03.146-0700	INFO	template/load.go:100	template with name 'metricbeat' loaded.
2019-07-16T06:31:03.146-0700	INFO	[index-management]	idxmgmt/std.go:289	Loaded index template.
2019-07-16T06:31:03.147-0700	INFO	[index-management]	idxmgmt/std.go:300	Write alias successfully generated.
Index setup finished.
8

@Michal_Pristas Hi Michal any idea why its not working?

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.