Strange issue with Elasticsearch while bulk indexing

Elastic Stack Elasticsearch
1

Hi I have bit of strange issue with elasticsearch, I have 30 shards 0 replicas and currently 14 million documents. I Bulk index using NEST like this:
public static bool MultiIndex(List objects) where T : someclass
{
bool a = false;
var bulkall = Program._elasticclient.BulkAll(objects, b =>
b.Index("indexname")
.BackOffRetries(2)
.BackOffTime("30s")
.RefreshOnCompleted()
.MaxDegreeOfParallelism(5)
.Size(10000));
bulkall.Subscribe(new BulkAllObserver(
onNext: (b) => { Console.WriteLine("inserted 10k"); },
onError: (e) => { throw e; },
onCompleted: () => a = true));
return a;
}
I have few servers running which process data and then index like that. I had no issue until now but after data grew, now that those servers multiindex few times elasticsearch becomes unresponsive and unusable for a while. How could I resolve this?

p.s each time I call MultiIndex I pass around 10-30k or so documents. each document is around 2kb
p.ss I gave it 16gb java heap.

2

I would recommend sending smaller batches. Aim for around 5 MB per request.

1 Like
3

I set refresh_interval to -1 before bulk indexing and then to 1s after I am done. Problem seems fixed. :smiley:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.