I'm using the elasticsearch:1.7.5 docker image, and setting up a new development environment. In importing data from production, I'm using bulk writes, retrieving 1000 documents at a time.
The problem is this. I can get 1000 documents in, and have the ES fail with a Java OOM error. As I understand it, the docker image has 2gb memory by default, and the default JVM settings for ES are 1gb. So, that should be good.
The documents I'm importing vary in size. There are a few large ones, and many small ones. I've taken to writing the bulk in batches of 10 documents (pretty small), which is taking forever.. which helps. But, it can still OOM error going this slow.
In the node stats, the heap used shows 84%. There is 0 field_data in the node. The total stored is around 500mb.
I'm sure there's something about how we have done our mapping.. that makes this painful. But, OOM'ing out on less than 1000 documents just seems wrong.
Am I doing something wrong here?