Filebeats is failing to perform the bulk index

[dubul]

File beat error:
ERR Failed to perform any bulk index operations: Post http://dt1es01p02:9200/_bulk: net/http: request canceled (Client.Timeout exceeded while awaiting headers)

Elastic search log:
[DEBUG][action.admin.indices.create] [dt1es01p02] [bit-logs-2016.12.30] failed to create
ProcessClusterEventTimeoutException[failed to process cluster event (create-index [bit-logs-2016.12.30], cause [auto(bulk api)]) within 1m]
at org.elasticsearch.cluster.service.InternalClusterService$2$1.run(InternalClusterService.java:349)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

I'm not sure about this error, Is this something I need to change the config file in elasticsearch or filebeat.yaml?
How could I fix this issue? I've tried changing the bulk index,timeout, flush interval but no use

[ruflin]

It seems like the index request takes too much time. Is your elasticsearch instance under heavy load? How many log events do you try to push per second?

[dubul]

@ruflin how to manage the filebeat to send this many logevents per second? Is there any config I need to make change. Now filebeat.yaml and elasticsearch.yaml is working with default settings

[ruflin]

First you should share some more information about your setup. What does your ES cluster look like? How many log lines do you have per second? General memory / CPU? ...

[dubul]

I have 3 elastic search servers with 8GB ram and 2 CPUs each.

[ruflin]

Can you share the number of log events you have per second and perhaps the size of each event?

[steffens (Steffen Siering)]

Reading the elasticsearch logs, it looks like the issue is with elasticsearch itself, due to creating an index takes longer then 1m.

If problem is with elasticsearch, changing filebeat.yml won't help. Have you tried to create an index via curl?