Index creation issues after removing ILM Policy

Elastic Stack Elasticsearch
1

Hi,
I have an issue regarding writing to a daily metricbeat index after removing ILM policy from index template. Below are the steps i followed:

  1. Created ILM policy to delete the index after 7 days (no rollup, cold, warm phases).
  2. Attached to a metricbeat index template which is used to create a daily index.
  3. After finding few issues (after few days) and assuming i configured ILM policy in a wrong way, i deleted the ILM policy mapping on index template and the policy too.
  4. Even after removing the ILM policy traces on index template, i am still seeing the same issue with the new index which is it is getting created but it is not getting written (the sizeof the index stays at 283 bytes and health says green). I suspect ILM policy is still taking care of the new indices or the metricbeat is failing to write to the index due to heavy queue size? Can someone help me understand what's going on here?

Thanks,
Sankeerth.

2

What do your Metricbeat logs show?

3

2020-08-12T13:23:03.299-0500 WARN elasticsearch/client.go:520 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfc52ac18dc78938, ext:885573859183401, loc:(*time.Location)(0x2950620)}, Meta:common.MapStr(nil), Fields:common.MapStr{"metricset":common.MapStr{"name":"cpu", "module":"system"}, "system":common.MapStr{"cpu":common.MapStr{"iowait":common.MapStr{"pct":0}, "idle":common.MapStr{"pct":3.9359}, "irq":common.MapStr{"pct":0}, "steal":common.MapStr{"pct":0}, "nice":common.MapStr{"pct":0}, "total":common.MapStr{"pct":0.0641}, "cores":4, "system":common.MapStr{"pct":0.0391}, "softirq":common.MapStr{"pct":0}, "user":common.MapStr{"pct":0.025}}}, "beat":common.MapStr{"name":"HOSTNAME", "hostname":"HOSTNAME", "version":"6.4.3"}, "host":common.MapStr{"name":HOSTNAME}}, Private:interface {}(nil)}, Flags:0x0} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse","caused_by":{"type":"illegal_argument_exception","reason":"Field [pct] misses required parameter [scaling_factor]"}}
2020-08-12T13:23:03.299-0500 WARN elasticsearch/client.go:520 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfc52ac18dc78938, ext:885573859183401, loc:(*time.Location)(0x2950620)}, Meta:common.MapStr(nil), Fields:common.MapStr{"system":common.MapStr{"network":common.MapStr{"in":common.MapStr{"packets":0x896b5c, "errors":0x0, "dropped":0xeb, "bytes":0xf64ba230}, "out":common.MapStr{"dropped":0x0, "packets":0x9d3c72, "bytes":0xf10830be, "errors":0x0}, "name":"Ethernet0"}}, "metricset":common.MapStr{"rtt":1001, "name":"network", "module":"system"}, "beat":common.MapStr{"name":"HOUDVDAPI2", "hostname":"", "version":"6.4.3"}, "host":common.MapStr{"name":""}}, Private:interface {}(nil)}, Flags:0x0} (status=400): {"type":"illegal_argument_exception","reason":"Rejecting mapping update to [metricbeat-6.4.3-2020.08.12] as the final mapping would have more than 1 type: [_doc, doc]"}

4

That would be why, it's not related to ILM it's something else you have configured.
Are you using Logstash?

5

Yes, i use logstash as output and the ES is 0n 7.6.2 where as metricbeat is on 6.4.3.

6

What does your Logstash output look like?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.