Elasticsearch server crashing

Elastic Stack Elasticsearch
1

Hello,

We have two Elasticsearch server which are configured as the Master-slave, one server keeps getting crash.We have recently upgraded the Elasticsearch server 5.1 to 7.1.1 on our client, It is giving the following error

>     [2021-03-01T02:22:46,544][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [VPRODGRMQ01] fatal error in thread [elasticsearch[VPRODGRMQ01][flush][T#317]], exiting
>     java.lang.StackOverflowError: null
>     	at org.elasticsearch.common.io.stream.ReleasableBytesStreamOutput.ensureCapacity(ReleasableBytesStreamOutput.java:70) ~[elasticsearch-7.1.1.jar:7.1.1]
>     	at org.elasticsearch.common.io.stream.BytesStreamOutput.writeByte(BytesStreamOutput.java:72) ~[elasticsearch-7.1.1.jar:7.1.1]
>     	at org.elasticsearch.common.io.stream.StreamOutput.write(StreamOutput.java:469) ~[elasticsearch-7.1.1.jar:7.1.1]
>     	at org.elasticsearch.transport.CompressibleBytesOutputStream.writeByte(CompressibleBytesOutputStream.java:80) ~[elasticsearch-7.1.1.jar:7.1.1]
>     	at org.elasticsearch.common.io.stream.StreamOutput.writeBoolean(StreamOutput.java:442) ~[elasticsearch-7.1.1.jar:7.1.1]
>     	at org.elasticsearch.common.io.stream.StreamOutput.writeOptionalString(StreamOutput.java:322) ~[elasticsearch-7.1.1.jar:7.1.1]
>     	at org.elasticsearch.common.io.stream.StreamOutput.writeException(StreamOutput.java:950) ~[elasticsearch-7.1.1.jar:7.1.1]
>     	at org.elasticsearch.common.io.stream.StreamOutput.writeException(StreamOutput.java:983) ~[elasticsearch-7.1.1.jar:7.1.1]
>     	at org.elasticsearch.ElasticsearchException.writeStackTraces(ElasticsearchException.java:725) ~[elasticsearch-7.1.1.jar:7.1.1]
>     	at org.elasticsearch.common.io.stream.StreamOutput.writeException(StreamOutput.java:985) ~[elasticsearch-7.1.1.jar:7.1.1]
>     	at org.elasticsearch.common.io.stream.StreamOutput.writeException(StreamOutput.java:983) ~[elasticsearch-7.1.1.jar:7.1.1]
>     	at org.elasticsearch.ElasticsearchException.writeStackTraces(ElasticsearchException.java:725) ~[elasticsearch-7.1.1.jar:7.1.1]

Adding gist for the full stacktrace of the exception
Error Stack

Everytime it crashes we have manually restart it again.

We have not been able to identify why is it happening, there is other error in the logs which is about not able to access the translog files, but they does not seems to be connected to the crash.
Any idea of what could be causing these crashes?

Thanks in advance for your answers!
Thanks,
JPross

2

Welcome to our community! :smiley:
There's no such thing as a slave in Elasticsearch, you would likely be referring to master-eligible nodes.

Is there more to the Elasticsearch log prior to this?
What is the output from the _cluster/stats?pretty&human API?

3

Hi Warkolm,
Thanks for the reply, sorry that was the term used by our IT to describe the infra for the Elasticsearch servers.

Here are the log file and the response from the _cluster API.
Elastic Log and cluster info

The crashing error is coming at '2021-03-01T02:22:46,544' on line 207

4

7.1.1 has been EOL for quite some time already. Can you reproduce this in a supported version? This looks like a bug, but quite possibly one that has since been fixed.

5

Hi,
Thanks for the reply. I can't reproduce it on the supported version as it is our clients Production environment. And I haven't seen this error in our local or the UAT environment.
Can you point us why this error might be occurring.

6

No, sorry, I don't even have a development environment for such an old version any more.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.