Hi I have bit of strange issue with elasticsearch, I have 30 shards 0 replicas and currently 14 million documents. I Bulk index using NEST like this:
public static bool MultiIndex(List objects) where T : someclass
{
bool a = false;
var bulkall = Program._elasticclient.BulkAll(objects, b =>
b.Index("indexname")
.BackOffRetries(2)
.BackOffTime("30s")
.RefreshOnCompleted()
.MaxDegreeOfParallelism(5)
.Size(10000));
bulkall.Subscribe(new BulkAllObserver(
onNext: (b) => { Console.WriteLine("inserted 10k"); },
onError: (e) => { throw e; },
onCompleted: () => a = true));
return a;
}
I have few servers running which process data and then index like that. I had no issue until now but after data grew, now that those servers multiindex few times elasticsearch becomes unresponsive and unusable for a while. How could I resolve this?
p.s each time I call MultiIndex I pass around 10-30k or so documents. each document is around 2kb
p.ss I gave it 16gb java heap.
