ElasticSearch stops responding after reaching a certain number of "fails to parse query" errors

Elastic Stack Elasticsearch
6

We shared some information by email, including the hot threads output and a diagnostics dump and it looks like the stuck node has many threads, including all of its network threads, busy writing to logs:

    0.0% (0s out of 500ms) cpu usage by thread 'elasticsearch[2fFs0rv][transport_client_boss][T#2]'
     10/10 snapshots sharing following 72 elements
       java.io.FileOutputStream.writeBytes(Native Method)
       java.io.FileOutputStream.write(FileOutputStream.java:326)
       java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
       java.io.PrintStream.write(PrintStream.java:480)
       org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.java:53)
       org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStreamManager.java:252)
       org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager.java:284)
       org.apache.logging.log4j.core.appender.OutputStreamManager.drain(OutputStreamManager.java:341)
       org.apache.logging.log4j.core.layout.TextEncoderHelper.drainIfByteBufferFull(TextEncoderHelper.java:147)
       org.apache.logging.log4j.core.layout.TextEncoderHelper.encodeAsMuchAsPossible(TextEncoderHelper.java:125)
       org.apache.logging.log4j.core.layout.TextEncoderHelper.encode(TextEncoderHelper.java:109)
       org.apache.logging.log4j.core.layout.TextEncoderHelper.encodeText(TextEncoderHelper.java:89)
       org.apache.logging.log4j.core.layout.StringBuilderEncoder.encodeSynchronized(StringBuilderEncoder.java:86)
       org.apache.logging.log4j.core.layout.StringBuilderEncoder.encode(StringBuilderEncoder.java:61)
       org.apache.logging.log4j.core.layout.StringBuilderEncoder.encode(StringBuilderEncoder.java:32)

I've suggested toning down the logging to see if that helps.

diff -U3 -r config-default/log4j2.properties config-0/log4j2.properties
--- config-default/log4j2.properties	2017-01-24 19:50:32.000000000 +0000
+++ config-0/log4j2.properties	2019-03-07 11:13:08.000000000 +0000
@@ -2,7 +2,7 @@

 # log action execution errors for easier debugging
 logger.action.name = org.elasticsearch.action
-logger.action.level = debug
+logger.action.level = info

 appender.console.type = Console
 appender.console.name = console