Filebeat unable to send data to elastic search and shows error as below.
May 18 23:37:03 filebeat_host filebeat[18405]: 2020-05-18T23:37:03.595+0530 ERROR pipeline/output.go:121 Failed to publish events: Post https://es_node1:9200/_bulk: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
Similar error for the other two-node as well.
On Es side we are getting some error as below,
[2020-05-18T18:13:47,864][INFO ][o.e.x.i.IndexLifecycleRunner] [master.elasticsearch.node] policy [filebeat] for index [test_dev-14] on an error step due to a transitive error, moving back to the failed step [check-rollover-ready] for execution. retry attempt [287]
[2020-05-18T18:13:47,866][ERROR][o.e.x.i.IndexLifecycleRunner] [master.elasticsearch.node] policy [filebeat] for index [index_Test2] failed on step [{"phase":"hot","action":"rollover","name":"check-rollover-ready"}]. Moving to ERROR step
java.lang.IllegalArgumentException: index.lifecycle.rollover_alias [ai_secops_filebeat_broadcomit] does not point to index [ai_secops_filebeat_broadcomit]
at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:104) [x-pack-core-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:152) [x-pack-ilm-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:328) [x-pack-ilm-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:266) [x-pack-ilm-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:183) [x-pack-core-7.6.2.jar:7.6.2]
at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:211) [x-pack-core-7.6.2.jar:7.6.2]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:830) [?:?]