Shard is not working due to _source field

[Butovoboy (Egor)]

Hello, guys!
I have a server with running ELK and other four servers with running filebeat on them. All these four filebeats send information directly to Elasticsearch on the first server.

One shard is not working and I get an error in Kibana:

My setup.template is similar in every filebeat.yml:

setup.template:
  name: "filebeat"
  pattern: "filebeat-*"
  settings:
    index.number_of_shards: 1
    index.number_of_replicas: 0
    index.codec: best_compression
    _source.enabled: true

So, _source is enabled everywhere, but It thinks that _source is not enabled. So, how to enable the _source field and to run this shard?

[warkolm (Mark Walkom)]

(We aren't all guys )

It's odd that it'd complain about a single shard, as mappings are consistent against all shards. Is there anything in the Elasticsearch logs at that time?

[Butovoboy (Egor)]

Thanks a lot for your reply!
Sorry for "guys", didn't want to hurt somebody

Yea, here are some errors in the elastic logs:

{"type": "server", "timestamp": "2022-01-10T07:03:53,413Z", "level": "ERROR", "component": "o.e.x.i.IndexLifecycleRunner", "cluster.name": "es-docker-cluster", "node.name": "es01", "message": "policy [filebeat] for index [filebeat-7.15.2-2022.01.10] failed on step [{\"phase\":\"hot\",\"action\":\"rollover\",\"name\":\"check-rollover-ready\"}]. Moving to ERROR step", "cluster.uuid": "Bej_vV1xSHaIbsoUYLrKcQ", "node.id": "HykhdOWRT1ipHsdhTDGP6Q" , 
"stacktrace": ["java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [filebeat-7.15.2] does not point to index [filebeat-7.15.2-2022.01.10]",
"at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:126) [x-pack-core-7.15.2.jar:7.15.2]",
"at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:176) [x-pack-ilm-7.15.2.jar:7.15.2]",
"at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:365) [x-pack-ilm-7.15.2.jar:7.15.2]",
"at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:303) [x-pack-ilm-7.15.2.jar:7.15.2]",
"at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:184) [x-pack-core-7.15.2.jar:7.15.2]",
"at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:217) [x-pack-core-7.15.2.jar:7.15.2]",
"at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) [?:?]",
"at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]",
"at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]",
"at java.lang.Thread.run(Thread.java:833) [?:?]"] }

But it seems to me that this error has been occurred some time after the shard has broken.

This is my another problem because I wrote in all filebeat.yml:

setup.ilm.enabled: false
ilm.enabled: false

But ilm is still running and I can't disable it throught GUI.

I didn't think that these two problems are connected to each other. So, no more errors do exist in Elasticsearch logs.