Strange issue with Elasticsearch while bulk indexing

Hi I have bit of strange issue with elasticsearch, I have 30 shards 0 replicas and currently 14 million documents. I Bulk index using NEST like this:
public static bool MultiIndex(List objects) where T : someclass
bool a = false;
var bulkall = Program._elasticclient.BulkAll(objects, b =>
bulkall.Subscribe(new BulkAllObserver(
onNext: (b) => { Console.WriteLine("inserted 10k"); },
onError: (e) => { throw e; },
onCompleted: () => a = true));
return a;
I have few servers running which process data and then index like that. I had no issue until now but after data grew, now that those servers multiindex few times elasticsearch becomes unresponsive and unusable for a while. How could I resolve this?

p.s each time I call MultiIndex I pass around 10-30k or so documents. each document is around 2kb I gave it 16gb java heap.

I would recommend sending smaller batches. Aim for around 5 MB per request.

1 Like

I set refresh_interval to -1 before bulk indexing and then to 1s after I am done. Problem seems fixed. :smiley:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.