Filebeat Index Lifecycle Management Issue (No Data)

Elastic Stack Elasticsearch
1

My filebeat index seems to be corrupted and will not take new ingestion data. While reviewing the logs I noticed the life cycle error below over and over in the logs. The filebeat index is the only index that seems to have the issue. (Metricbeat, Functionbeat etc... are Okay) I have tried deleting the index and removing the lifecycle rule however that does not seem to resolve the issue. I am running version 7.0.0 along with Logstash for log ingestion.

Any help would be much appreciated!

#  - Creating and Starting rollup jobs will no longer be allowed.
#  - Stopping/Deleting existing jobs, RollupCaps API and RollupSearch continue to function.
[es/i-0/es.log] [2019-05-06T12:15:12,688][ERROR][org.elasticsearch.xpack.indexlifecycle.IndexLifecycleRunner] [instance-0000000000] policy [filebeat-7.0.0] for index [filebeat-7.0.0-2019.05.06] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step
java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [filebeat-7.0.0] does not point to index [filebeat-7.0.0-2019.05.06]
              at org.elasticsearch.xpack.core.indexlifecycle.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:92) [x-pack-core-7.0.0.jar:7.0.0]
              at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:133) [x-pack-ilm-7.0.0.jar:7.0.0]
              at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:270) [x-pack-ilm-7.0.0.jar:7.0.0]
              at org.elasticsearch.xpack.indexlifecycle.IndexLifecycleService.triggered(IndexLifecycleService.java:213) [x-pack-ilm-7.0.0.jar:7.0.0]
              at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:168) [x-pack-core-7.0.0.jar:7.0.0]
              at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:196) [x-pack-core-7.0.0.jar:7.0.0]
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_144]
              at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_144]
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_144]
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_144]
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_144]
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_144]
              at java.lang.Thread.run(Thread.java:748) [?:1.8.0_144]
2

Do you mind sharing the index mapping for filebeat-7.0.0-2019.05.06

and sharing the template for your filebeat indices? the filebeat-7.0.0 alias should be associated with these indices

3

for more information on configuration: https://www.elastic.co/guide/en/beats/filebeat/7.0/ilm.html

4

Please see the attached files….

(Attachment filebeat-7.0.0-2019.05.06-mapping is missing)

(Attachment filebeat-7.0.0-template is missing)

5

I am not sure if you received the attachments as the contents of the mapping and template files exceed the character limit of the reply posting. If you have not received the attached files please let me know how you would like me to send the contents of the files. Thanks!

6

I did not. you can send them to my email tal (at) elastic.co if you'd like. I can take a look at them there

7

Sent the logs, I looked and the alias and the name seems to match up however I see this error message in the index management console only for the filebeat indexes.

filebeat-index-error
filebeat-index-error935×279 24.6 KB

8

thanks

To fix this, the index you are writing to should have a filebeat-7.0.0 alias set as the write-index

more information on bootstrapping can be found here: https://www.elastic.co/guide/en/elasticsearch/reference/7.0/getting-started-index-lifecycle-management.html#ilm-gs-apply-policy

updating the index's alias should resolve this.

It is not clear to me why Filebeat did not do this on your behalf correctly. Will look into it and get back to you!

9

following up here:

did you ever delete any filebeat indices while you were testing?

might be related: https://github.com/elastic/elasticsearch/issues/41699

10

Yes, I have deleted several filebeat indexes along with deleting the lifecycle rule and trying to create it again in an attempt to resolve the issue.

11

I tried to enable the index alias however has not resolved the issue.

12

It looks like this might be a bug with the index life cycle rules in Kibana. I was able to resolve the issue by removing the indexes from the life cycle rules after creating a life cycle rule with the same name. I think originally what happened is that I created a life cycle rule prior and deleted it when it had indexes that were being managed by the life cycle rule. Is this a potential bug or a known issue with life cycle rules managing indexes?

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.