I am so relieved someone else ran into this. I thought I was somehow seeing that bug from 5.6 reappear. I was pretty confused when I ran into this and ended up using that workaround from the bug where I set the parent limit_size to, uh, 8 exabytes . While this did not feel great, the problem went away and I was able to resume indexing documents.
I am also using JDK12 and G1GC with the defaults on a 8GB heap. I'm only bulk indexing docs in a single thread with some occasional mget's. No searches or aggs. GC's are very effective when they run, taking the heap from 75% down to 25%.
It seems like sending a bulk insert when G1GC's heap is near 75% temporarily puts it over the edge, which I think is what Daniel is suggesting here:
I will try reverting to CMS first, and then try setting indices.breaker.total.use_real_memory
to false
if it's still an issue.
"breakers" : {
"request" : {
"limit_size_in_bytes" : 4986608025,
"limit_size" : "4.6gb",
"estimated_size_in_bytes" : 0,
"estimated_size" : "0b",
"overhead" : 1.0,
"tripped" : 0
},
"fielddata" : {
"limit_size_in_bytes" : 3324405350,
"limit_size" : "3gb",
"estimated_size_in_bytes" : 3440,
"estimated_size" : "3.3kb",
"overhead" : 1.03,
"tripped" : 0
},
"in_flight_requests" : {
"limit_size_in_bytes" : 8311013376,
"limit_size" : "7.7gb",
"estimated_size_in_bytes" : 1860,
"estimated_size" : "1.8kb",
"overhead" : 2.0,
"tripped" : 0
},
"accounting" : {
"limit_size_in_bytes" : 8311013376,
"limit_size" : "7.7gb",
"estimated_size_in_bytes" : 51488284,
"estimated_size" : "49.1mb",
"overhead" : 1.0,
"tripped" : 0
},
"parent" : {
"limit_size_in_bytes" : 9223372036854775807,
"limit_size" : "8192pb",
"estimated_size_in_bytes" : 6715789312,
"estimated_size" : "6.2gb",
"overhead" : 1.0,
"tripped" : 0
}
}```