Elasticsearch 6.0 bulk write is slower in a cluster but fast in single node setup

Elasticsearch assumes all data nodes to be equal, so the fact that you have one large and 2 small will mean that the the smaller ones will work much harder than the larger one and limit performance.

The main point of having a cluster is usually to avoid single points of failure in order to increase stability, availability and resiliency. A production setup with a single dedicated master node is therefore not recommended. Just because you can have specialised node types does not mean that you should. I would recommend that you instead setup a basic 3 node cluster where all nodes are on the same type of hardware and are all master eligible and hold data. Make sure that you set minimum_master_nodes to 2 to avoid split brain scenarios.

How have you arrived at this value? That is a lot of shards to index into, which may lead to bad indexing performance and possibly also bulk rejections.

If you are not updating your data, you may want to consider instead using time-based indices, as this generally is more efficient. You may also want to read this blog post on shards and sharding practices.

1 Like