# Jre crash after running for days or hours
**URL:** https://discuss.elastic.co/t/jre-crash-after-running-for-days-or-hours/262439
**Category:** Elasticsearch
**Created:** [January 28, 2021, 2:08am UTC](https://discuss.elastic.co/t/jre-crash-after-running-for-days-or-hours/262439 "2021-01-28T02:08:40Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@billhong-just](https://discuss.elastic.co/u/billhong-just)
#### Post date: [January 28, 2021, 2:08am UTC](https://discuss.elastic.co/t/jre-crash-after-running-for-days-or-hours/262439/1 "2021-01-28T02:08:40Z")
Hi community,
My single node elasticsearch cluster crashed after running for days or hours.
And here are relevant information and logs.
#### Elasticsearch version (bin/elasticsearch --version):
7.10.2
#### Plugins installed:
I install elasticsearch following this [doc](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/deb.html).
No other plugin installed.
#### JVM version (java -version):
- JRE version: OpenJDK Runtime Environment AdoptOpenJDK (15.0.1+9) (build 15.0.1+9)
- Java VM: OpenJDK 64-Bit Server VM AdoptOpenJDK (15.0.1+9, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
#### OS version (uname -a if on a Unix-like system):
Ubuntu 20.04.1 LTS (GNU/Linux 5.4.0-42-generic x86\_64)
Linux sw-vwordpress01 4.15.0-91-generic #92-Ubuntu SMP Fri Feb 28 11:09:48 UTC 2020 x86\_64 x86\_64 x86\_64 GNU/Linux
#### Description of the problem including expected versus actual behavior:
I am running a single node elasticsearch cluster for [elastic observability](https://www.elastic.co/guide/en/observability/current/observability-introduction.html).
After running for hours or days, the cluster crash.
#### Steps to reproduce:
I use [apm-agent-dotnet v1.6.1](https://github.com/elastic/apm-agent-dotnet/releases/tag/1.6.1) to send apm transaction & metrics to APM server.
The APM server stay on the same server which host elasticsearch single node cluster.
After running for hours or days, the cluster crash.
And then it produce a `hs_err_pidXXXXX.log` in `/var/log/elasticsearch` directory.
```auto
# Problematic frame:
# J 14564 c2 org.apache.lucene.codecs.DocValuesConsumer$SortedNumericDocValuesSub.nextDoc()I (8 bytes) @ 0x00007f650c700669 [0x00007f650c700620+0x0000000000000049]
```
#### Provide logs (if relevant):
- [hs\_err\_pid13402.log](https://github.com/elastic/elasticsearch/files/5865160/hs_err_pid13402.log)
- [hs\_err\_pid1143.log](https://github.com/elastic/elasticsearch/files/5865161/hs_err_pid1143.log)
- Execute `sudo systemctl status elasticsearch` show the following messages
---
### Author:  [@billhong-just](https://discuss.elastic.co/u/billhong-just)
#### Post date: [January 28, 2021, 2:10am UTC](https://discuss.elastic.co/t/jre-crash-after-running-for-days-or-hours/262439/2 "2021-01-28T02:10:22Z")
Yesterday my single node elasticsearch cluster crash again, then I dump relevant logs.
But I cannot tell which hardware caused it from the output of `dmesg`.
Could anyone help me to point out where the problem is?
I also opend a issue [here](https://github.com/elastic/elasticsearch/issues/67882).
#### Provide logs (if relevant):
- [hs\_err\_pid1144.log](https://github.com/elastic/elasticsearch/files/5877141/hs_err_pid1144.log)
- [dmesg\_2021-01-27.log](https://github.com/elastic/elasticsearch/files/5877142/dmesg_2021-01-27.log)
- Execute `sudo systemctl status elasticsearch` show the following messages
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [January 28, 2021, 2:21am UTC](https://discuss.elastic.co/t/jre-crash-after-running-for-days-or-hours/262439/3 "2021-01-28T02:21:47Z")
Welcome to our community! 😃
What's in the Elasticsearch log, usually under `/var/log/elasticsearch/`?
---
### Author:  [@billhong-just](https://discuss.elastic.co/u/billhong-just)
#### Post date: [January 28, 2021, 2:34am UTC](https://discuss.elastic.co/t/jre-crash-after-running-for-days-or-hours/262439/4 "2021-01-28T02:34:26Z")
Hi, mark
Here are logs under `/var/log/elasticsearch/`.
> **[logs - Google Drive](https://drive.google.com/drive/folders/1bYskRNun0PDEZad-1i2F3gJAkQU3_vb8?usp=sharing)**
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [January 28, 2021, 2:50am UTC](https://discuss.elastic.co/t/jre-crash-after-running-for-days-or-hours/262439/5 "2021-01-28T02:50:32Z")
Can you please post them if they are not too big, or use gist/pastebin/etc.
---
### Author:  [@billhong-just](https://discuss.elastic.co/u/billhong-just)
#### Post date: [January 28, 2021, 3:08am UTC](https://discuss.elastic.co/t/jre-crash-after-running-for-days-or-hours/262439/6 "2021-01-28T03:08:30Z")
Hi, mark
Here is the gist.
Please click the [link](https://gist.github.com/billhong-just/402962a3fd1f289646b8c350422afa58) to see total **18** logs.
>
>
> There are more than three files. show original
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [January 28, 2021, 8:21am UTC](https://discuss.elastic.co/t/jre-crash-after-running-for-days-or-hours/262439/7 "2021-01-28T08:21:59Z")
```auto
[Mon Jan 25 22:05:40 2021] node[1175]: segfault at 1 ip 0000000000000001 sp 00007ffee4ae5248 error 14 in node[400000+204e000]
[Mon Jan 25 22:05:40 2021] Code: Bad RIP value.
[Tue Jan 26 20:48:05 2021] traps: node[5038] trap invalid opcode ip:17aad89 sp:7fc92b992898 error:0 in node[400000+204e000]
```
This looks like bad hardware, although more likely bad RAM or CPU rather than storage. Does this reproduce on a different machine?
---
### Author:  [@billhong-just](https://discuss.elastic.co/u/billhong-just)
#### Post date: [February 2, 2021, 3:04am UTC](https://discuss.elastic.co/t/jre-crash-after-running-for-days-or-hours/262439/8 "2021-02-02T03:04:56Z")
I have moved my single node elasticsearch cluster to a different machine, and it runs healthily for about 4 days.
And there is no hardware error message in the output of `dmesg`.
I will keep watching and report here.
Thanks for your help.
---
### Author:  [@billhong-just](https://discuss.elastic.co/u/billhong-just)
#### Post date: [February 22, 2021, 1:19am UTC](https://discuss.elastic.co/t/jre-crash-after-running-for-days-or-hours/262439/9 "2021-02-22T01:19:19Z")
Hi community,
It has been about 3 weeks since my last report.
And my elasticsearch cluster stays healthy since then.
We can confirm the problem is about bad hardware.
Thank you all.
😉
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 22, 2021, 1:19am UTC](https://discuss.elastic.co/t/jre-crash-after-running-for-days-or-hours/262439/10 "2021-03-22T01:19:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.