Hey folks. We have been doing some ES benchmarks focused on indexing large amounts of small documents. These tests have been done on single nodes and with small clusters of just two or three nodes, and one thing we are a concerned about is that when we go from a single-node to a (small) multi-node setup, we don't see indexing performance improve. We're worried that we're doing something wrong. (Maybe we just need to keep adding nodes?)
Here's our setup:
- Elasticsearch 1.7.1
- RAID 0 of a lot of 10K disks
- Plenty of CPU (2 CPUs, 8 cores each, with Hyper-Threading on, so 2x2x8 = 32 cores)
- 128 GB RAM
- JVM heap capped to 30 GB
- Not much else running on the machine
- Java 7
- mlockall is on
- Max fd is 64,000
- Upped index.refresh_interval to 30s
- Turned off index throttling
- Primary shards set to 2 * number of nodes
- 1 replica
- 1 merge thread and max merge count of 6 (not sure what's best here)
- Inserting with Bulk API
- Each bulk insert is about 5 MB in size
- Each document is about 1 KB
- Documents contain data you'd see from a webserver log (like URL, status code)
We can do a sustained ~50K insert/second on a single node. Interestingly, we do not seem to be I/O bound at all. When benchmarking our disk array separately, we can do about 1 GB/s and 1K writes/s. With ES, we see spikes to the system up to 250 MB/s and 350 MB/s to disk every few seconds. Writes per second is spikey but below 600.
We've tried our same test on a VM with half the CPU and RAM resources and, somewhat surprisingly to us, we again reached around ~50K inserts/second, so I am concerned our setup is hitting some kind of limitation. Nothing is immediately apparent, though. Any thoughts or pointers here to understanding more about any limitations we might be hitting?
More to the point of this post, when we try the same test on a two or three node cluster, we don't see any improvement in indexing speed (it gets a little worse).
With multi-node tests we are just doing round-robin bulk API inserts, nothing fancy. Replication is set to 1 so there will be some overhead incurred there, but I assumed not so much as to prevent gains in indexing perf. We are not using a separate dedicated master.
I'm a little concerned we simply have some basic aspect of our setup or indexing incorrect but nothing stands out. Curious if anyone else has run into anything similar or has any pointers for helping us diagnose. Also let me know if there are more specific I can provide. Thanks!