Beats write_index causes single hotspots


I'm using beats to ship most of my data to elasticsearch. I'm using the suggested ILM in a hot-warm-cold setup, which works fine.

The biggest problem I see with the way beats utilizes the write_indices with aliases, is that (even if I have 6 hot nodes), all of the current ingest traffic is going to that single hot node.

I have filebeat, metricbeat and heartbeat, so I can spread the load over these three write indices, but if I want to include more data in the filebeats, I'm not able to spread the load.

Any thoughts about how I can tackle this problem? Because I sometimes see bulk rejections (see

Ok, so after giving it some more thoughts, I just have to change the default number_of_shards from 1 to 3 (for example).

The "shrink" index option in the warm phase, will take care of the rest :slight_smile:.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.