This question is cross-posted from Amazon's Elasticsearch Forum. (link) I'm unsure whether it's an Elasticsearch or an AWS issue - but would welcome any help/feedback this community can offer.
Bulk indexing requests pointed at a AWS ES domain's VPC endpoint are not being round-robined to each of the data nodes for processing. This can most obviously be seen by looking at the instance health dashboard: https://imgur.com/a/wdhlkhk
Domain settings:
- 5 r5.xlarge.elasticsearch nodes
- 1 AZ
- General Purpose SSD (60GB per node)
- Isolated in a private subnet
Full settings: https://imgur.com/a/vFOrueC
Background
I have a process that uses the Elasticsearch Bulk API to load data. I am pointing the process at the Elasticsearch VPC endpoint. I assumed that there was a load balancer between this endpoint and the data nodes - but I am looking at the individual health of each instance and it's clear that the indexing operations are only being handled by one of the nodes.
Additional settings I have configured to accelerate indexing:
replicas: 0
shards: 1
index refresh interval: -1
translog flush threshold size: 1024MB
Data Instance Health: https://imgur.com/a/wdhlkhk
Index Settings: https://gist.github.com/thegrif/64c3d5f1b4465cf13d3ddeaee4d2b9a2
Overall Stats: https://gist.github.com/thegrif/a43ff8fcfd83eca254d4f9a90dfb57d0
Node Info: https://gist.github.com/thegrif/fa55e72f3e995bbb8ba44f3a2dc51a57
Node Stats: https://gist.github.com/thegrif/acae53ffbfe7a6f154b175332779817f
...