So I have attached our Hot Thread, https://gist.github.com/zsiddique/5cf5212582f87dd1cf8c9332c553d8c9, we are using an update script (Groovy right now) to handle updates. We need a script to update because we use nested data type and want to handle updates in that nested field.
We are seeing a handful of nodes have high CPU (mostly those with more primary shards) but other nodes are pretty ok (around 50%). We are indexing peak around 10k/s, around 8-9k/s normally.
We are running 12 data nodes, i3.2xl on AWS. I would expect we could push more documents per seconds.