Horizontal scaling of indexing

Elastic Stack Elasticsearch
#1

Hey folks. We have been doing some ES benchmarks focused on indexing large amounts of small documents. These tests have been done on single nodes and with small clusters of just two or three nodes, and one thing we are a concerned about is that when we go from a single-node to a (small) multi-node setup, we don't see indexing performance improve. We're worried that we're doing something wrong. (Maybe we just need to keep adding nodes?)

Here's our setup:

  • Elasticsearch 1.7.1
  • RAID 0 of a lot of 10K disks
  • Plenty of CPU (2 CPUs, 8 cores each, with Hyper-Threading on, so 2x2x8 = 32 cores)
  • 128 GB RAM
  • JVM heap capped to 30 GB
  • Not much else running on the machine
  • Java 7
  • mlockall is on
  • Max fd is 64,000
  • Upped index.refresh_interval to 30s
  • Turned off index throttling
  • Primary shards set to 2 * number of nodes
  • 1 replica
  • 1 merge thread and max merge count of 6 (not sure what's best here)
  • Inserting with Bulk API
  • Each bulk insert is about 5 MB in size
  • Each document is about 1 KB
  • Documents contain data you'd see from a webserver log (like URL, status code)

Results:

We can do a sustained ~50K insert/second on a single node. Interestingly, we do not seem to be I/O bound at all. When benchmarking our disk array separately, we can do about 1 GB/s and 1K writes/s. With ES, we see spikes to the system up to 250 MB/s and 350 MB/s to disk every few seconds. Writes per second is spikey but below 600.

We've tried our same test on a VM with half the CPU and RAM resources and, somewhat surprisingly to us, we again reached around ~50K inserts/second, so I am concerned our setup is hitting some kind of limitation. Nothing is immediately apparent, though. Any thoughts or pointers here to understanding more about any limitations we might be hitting?

More to the point of this post, when we try the same test on a two or three node cluster, we don't see any improvement in indexing speed (it gets a little worse).

With multi-node tests we are just doing round-robin bulk API inserts, nothing fancy. Replication is set to 1 so there will be some overhead incurred there, but I assumed not so much as to prevent gains in indexing perf. We are not using a separate dedicated master.

I'm a little concerned we simply have some basic aspect of our setup or indexing incorrect but nothing stands out. Curious if anyone else has run into anything similar or has any pointers for helping us diagnose. Also let me know if there are more specific I can provide. Thanks!

1 Like
#2

Glad to hear someone else seeing same problems we are seeing. We're using slightly different HW with similar results.

See my old post here:

Once we got past the testing harness setup, we were able to reproduce the slow indexing performance internally.

Our HW is:

Virident PCIe SSD card (config for performance) 1.8TB
64GB RAM
2x12 core Xeon (HT on, or equiv of 48) (5 physical bare metal nodes x 2 sets for faster testing of various parameters combination)
ES v1.7.2
JDK 8u60 (also tested with JDK7u51, JDK8u40)
Tested various maxheap from 16G to 31G.
mlockall on
max fd is 64K
refresh interval is -1
index.store.throttle.type: none
index.store.throttle.max_bytes_per_sec: 700mb
index.translog.flush_threshold_size: 1gb
indices.memory.index_buffer_size: 512mb
5 shards so we get 1 per node
no replica
various doc size from 1k to 16K
same data set on a RAMdisk so we always read same data via logstash file input
tested with 1 LS instance, 5 instances, 20 instances, etc.
Various bulk indexing sizes (100, 500, 1000, 5000, 10000, etc.).

Our conclusion is that I/O, CPU and memory are not the problem. We always hit a limit in how fast ES can index.

How are you ingesting data? Logstash? or your own client doing bulk insert? You can try increasing the number of instances feeding ES. We notice a slight increase in indexing speed, but it falls off after 10 concurrent LS instances into the 5 ES nodes.

#3

Hi tinle, thanks for the reply.

That we appear to not be hitting any resource limit (I/O, CPU, memory) is part of what I'm worried about (CPU limit is where we get closest I think). But the bigger concern is that adding nodes does not improve indexing performance...

We're not using Logstash here, we are doing our own bulk insert, and have also tested with really cranking up concurrent bulk requests, but the max we've been able to get is still around 50,000 docs per second w/ a doc size of ~1K.

#4

You have mentioned CPU, and RAM, but not the network. In my experiments with SSD nodes, I can saturate the network links.

If you hit 1Gbps, your network bandwith limit is probably exceeded.

There are many solutions:

  • switch off replica level while indexing
  • reduce shards (will also reduce transports)
  • use multiple NICs in ES config
  • channel bonding on OS level
  • upgrade network interface and router links to 10Gbps or better
#5

Also, you do not mention that with each new node, you have to add more capacity on client side to push enough data per time unit into the cluster.

#6

Yes, agree, network is not a problem however. We also monitor our network usage and they are definitely not saturated. We have pair of GigE bonded. As we add more ES nodes, we also add more LS nodes, as I mentioned, we tested 1, 5, 10, 15, 20, 25 and 50 LS instances going to just 5 ES nodes.

At peak, we see up to 2MB/s, and average 1MB/s during the several hours of testing, on our network bond0 interface.

We have a ticket open w/Elastic and have tried all various tuning parameters suggested. So it is good to hear from someone else other than us that is reporting the same problems.

#7

Hey jprante, thanks for the reply.

About our network setup: we have 10 Gbps connectivity between ES nodes.

In our multi-node tests, we appear to hit up to around 300 Mbps between ES nodes.

#8

Any updates with anyone on this front?

When I've got my single-node setup tuned best, I have experienced similar behavior, with one or more instances of ES on that node. The hardware that I am working with is similar to octal's, although I have slightly higher average document sizes.

Tinle, has your ticket with elastic been resolved to satisfaction?

#9