Yeah, it could be GC. The cycle at which this packet drop/zero indexing rate occurs is quite regular.
How do I check how many CPU cores ES has access to? I did not do any specific configurations, so I assume it can use all 96? I can see many, if not all, of the CPUs in use when running top. The average CPU usage is roughly 20-25%.
When I ran iostat, the %iowait is 0.03, and kB_wrtn/s is quite stable at 137K.
There is some improvement when I increased the number of shards from 2 to 4.
- When running 1 filebeat instance, the indexing rate went from 24K/s to 28K/s.
- For 2 filebeat instances, the total indexing rate went from 34K/s to 42K/s.
- But when the number of shards is kept constant (be it 2 or 4), increasing the number of filebeat instances reduces the indexing rate of a single instance. For example, at 4 shards, when there was only 1 filebeat, the indexing rate was 28K/s, but when there were 2 filebeats, the indexing rate of each filebeat dropped to 21K/s.
I've read that the recommended shard size is 10-50GB. I've kept my index size at 40GB, so I can at most have 4 shards (I did try with 8 shards, and only got 1% improvement in performance). Unless I increase my index size to, say, 100GB, then I can have up to 10 shards. Should I just increase my index size so I can parallelize more with more shards (yet at the same time, try not to have too many that degrades search performance)?
I'm trying to understand the benefits of having multiple nodes in a cluster. It seems like the number of shards is the main limiting factor for indexing performance. How does having more nodes help, as I can only write to 1 index each time, and can't increase the number of shards anymore (since it depends on index size) whether I have one node or 100 nodes? Or do the shards spread across different nodes such that it's not only writing to one node at a time?