Hi, I cannot reproduce the OOME stacktrace since I reindexed everything to
doc_values and now it is just slow, CPUs on 100% and no other obvious bottlneck like IO wait or GC chocking, probably more heap memory would help. We have 300 mils of documents in 100 indices on 3 nodes using 5 shards, each node is a c4.large instance, each ES has 2048MB ram. And default configuration, I'm not exactly sure whether field data cache would improve it radically.
I'm using ES 1.5.1
However I tried to set scroll batch size to 2,10, 40 to 200 which should roughly correspond to 0.1 to 5MB after multiplying the number of shards the request might hit. Not more... And it is still insanely slow, I played with all day and sorted scroll over 300mills of documents with just 4
doc_values field loaded would take hundreds of hours. It was scanning like 100 - 400 documents per second, not more.
I'm actually not sure if ec2 c4.large instance storage has any filesystem cache and how big is it. But it most probably does. It wouldn't make sense. We are using instance storages, not EBS.