Another case of slow bulk inserts on 2.3.1

Elastic Stack Elasticsearch
1

Hello!

We have a small ES cluster which we upgraded from 1.7.0 to 2.3.1 a few days ago, and now the system can't keep up with the bulk inserts we throw at it. I know there are more issues like this in the forum, but nothing talked about there helped.

An idea what we're doing:

  • = 2.000.000.000 documents in the cluster

  • about 200.000.000 inserts per day (an index per day (~400GB, 18 shards, 1 replica))
  • Google cloud engine with remote spinning disks(!)
  • only bulk inserts
  • very low query volume

This setup used to hold up fine with 1.7, even with the remote spinning disks in a cloud setup, to our surprise, and we have ran it for about two years now (adding machines as needed).

After the upgrade to 2.3.1 the bulk inserts can't keep up. I've tried playing with the bulk size and that didn't make a difference.
Some observations:

  • adding new machines doesn't really help. Looking at load graphs I see that it's usually a single machine which is at 90% for an hour or so, while the others are at a lower usage (60%). That switches to another machine, seemingly randomly.
  • /_stats/store is really slow. It takes between 15 and 60 seconds to load (Kopf uses this resource).
  • iostat, vmstat and similar tools don't show anything obvious. Disk IO is much lower than what we're allowed. 'wa' is mostly zero.
  • there are a dozen bulk clients, all trying to insert to all machines at the same time, so it's not the clients overloading a single machine
  • error logs are empty

Anyone any idea whan we can try? I've been looking at this forum, github issues, and the docs for the last few days, and nothing made any improvement.

Thanks!
Harmen
(edits: fixed docs per day estimate)

1 Like
2

Hello!

What kind of recommendation you've already used?

Which GC are you using?

3

Thanks for your reply!

I've been playing with the doc_values, but they seemed to be off for the indices converted from 1.7.
We've been trying to have only single / a few clients inserting, not all at the same time.
I've been playing with the bulk threads queue size.
Insert threads is 1, which is recommended.
There were some other discussions, which pointed to not relevant settings since we don't have many reads, don't remember which ones exactly.
We restarted machines which seemed to be overloaded, which only messed up the cluster (doh).

GC? We use the default as Debian installs it: /usr/bin/java -Xms12802m -Xmx12802m -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC -Dfile.encoding=UTF-8 -Djna.nosys=true

(about 50% of the 25GB of memory is used)

$ java -version
java version "1.8.0_31"
Java(TM) SE Runtime Environment (build 1.8.0_31-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.31-b07, mixed mode)

4

As for GC we have an issue with G1GC.
What about refresh_interval (increase to 30s or 60s) and translog.durablity (we get about 10-20% throughput increase with async but is less secure and can lead to data loss) ?

5

I've set refresh_interval to 60s for todays index. Will try the durability later.
How can we set the GC to the CMS GC?
Any idea about the reason of the slow /_stats/store endpoint?

We're now running with fewer clients inserting, and that seems to help.

We have deeply nested documents, 4 local disks, and home-made IDs (but they start with a timestamp).

6

Setting refresh_interval to 60s seems to help. Not all the time, but sometimes the cluster runs nicely with all systems at > 90% CPU.

7

Have you re-created all your data and pushed it over the ES API, or did you re-use 1.x indices with ES 2.x?

8

I let ES convert it (took 12 hours).

9

Maybe that's something to think about. I started from scratch with zero docs in ES 2.x and had no bulk insertion speed issues at all.

10

We're slowly moving to indices created after the upgrade, in 2.3.1, so we'll soon see!
(we still have lots of old data to insert first, though)

11

It's enabled by default. If you don't change it there is nothing to worry about.

Another thing is your home made IDs, I may be wrong but this can play role in one node overhead. Please check that documents spread by shards almost equally. And recheck this link this may help to improve indexing speed as ES 2.x check document ID for existence before inserting.

12

Maybe the IDs make things hard, but we didn't change that. So far doing less concurrent inserts seems to go better...