Without the verbose logs, I don't think that gets us any closer. It sounds like you were able to reproduce it though? Do you have step-by-step instructions that someone else could follow (ideally exact commands to run)?
In our case it turned out that an APM agent instrumenting the Elasticsearch node was the culprit, which didn't "know" about the move of the XContent class in ES 7.16.0 yet.
Sorry for the false alarm and thanks for looking into this! 
We're also running some APM-agent that instruments JVM-processes dynamically.
Regarding this issue I didn't do the connection to the APM-agent before.
Please take a look into these logs: Elasticsearch 7.16.2 verbose logs · GitHub
These are the last I could create/trigger.
The inital logs were from different nodes.
But Elasticsearch 7.16.1 crashing randomly - #21 by ppuschmann matches to these verbose-logs.
Should I look for something in (up-to-date) application-logs?
The only thing that stands out to me is /tmp/.instana/sensor-Elasticsearch-javaagent-1.1.59-29035-6754744723716598642.jar. I can't find a copy of that jar or the source code, but it looks like Instana injects code for the sake of monitoring? Maybe they have compiled against an old version of Elasticsearch? I'm just guessing though. Can you disable that and see if the problem is reproducible?
Hi Keith, I will try to disable Instana and do some tests again.
Everything is now looking fine.
So it was an issue of the APM-tool, not Elasticsearch 
Sorry.
2 Likes
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.