Job for elasticsearch.service failed because the control process exited with error code- VPS CentOs 7 Linux

[balu_thomas]

[root@--217-28 Packages]# systemctl status elasticsearch.service -l
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Sun 2019-12-01 06:38:32 EST; 1min 35s ago
Docs: http://www.elastic.co
Process: 20082 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
Main PID: 20082 (code=exited, status=1/FAILURE)
CGroup: /system.slice/elasticsearch.service

Dec 01 06:38:20 .-217-28.webhostbox.net systemd[1]: Starting Elasticsearch...
Dec 01 06:38:21 .-217-28.webhostbox.net elasticsearch[20082]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Dec 01 06:38:32 .-217-28.webhostbox.net systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Dec 01 06:38:32 .-217-28.webhostbox.net systemd[1]: Failed to start Elasticsearch.
Dec 01 06:38:32 .-217-28.webhostbox.net systemd[1]: Unit elasticsearch.service entered failed state.
Dec 01 06:38:32 .-217-28.webhostbox.net systemd[1]: elasticsearch.service failed.!

Journalctl Output:
[root@--217-28 Packages]# journalctl -xe
Dec 01 06:47:55 --217-28.webhostbox.net sshd[20858]: Disconnected from 41.77.145.34 port 33624 [preauth]
Dec 01 06:48:10 --217-28.webhostbox.net spamd[1893]: spamd: connection from localhost [::1]:53890 to port 783, fd 5
Dec 01 06:48:10 --217-28.webhostbox.net dovecot[1157]: pop3-login: Aborted login (no auth attempts in 0 secs): user=<>, rip=127.0.0.1, lip=127.0.0.1, secured, session=<4TeOCaOY
Dec 01 06:48:10 --217-28.webhostbox.net spamd[1459]: prefork: child states: II
Dec 01 06:48:11 --217-28.webhostbox.net dovecot[1157]: lmtp(20921): Connect from local
Dec 01 06:48:11 --217-28.webhostbox.net dovecot[1157]: lmtp(20921): Disconnect from local: Client has quit the connection (state=READY)
Dec 01 06:48:11 --217-28.webhostbox.net dovecot[1157]: imap-login: Login: user=<__cpanel__service__auth__imap__m_yr_xfk947wskrj>, method=PLAIN, rip=127.0.0.1, lip=127.0.0.1, mp
Dec 01 06:48:11 --217-28.webhostbox.net dovecot[1157]: imap(__cpanel__service__auth__imap__m_yr_xfk947wskrj)<20929><ryWXCaOY2IJ/AAAB>: Logged out in=11, out=490, bytes=11/490
Dec 01 06:48:11 --217-28.webhostbox.net pure-ftpd[20935]: (?@127.0.0.1) [INFO] New connection from 127.0.0.1
Dec 01 06:48:11 --217-28.webhostbox.net pure-ftpd[20935]: (?@127.0.0.1) [INFO] __cpanel__service__auth__ftpd__Q7HnwcDJ3oSibEPe is now logged in
Dec 01 06:48:11 --217-28.webhostbox.net pure-ftpd[20935]: (__cpanel__service__auth__ftpd__Q7HnwcDJ3oSibEPe@127.0.0.1) [INFO] Logout.
Dec 01 06:48:11 --217-28.webhostbox.net sshd[20882]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=183.201.252.68 user=root
Dec 01 06:48:11 --217-28.webhostbox.net sshd[20882]: pam_succeed_if(sshd:auth): requirement "uid >= 1000" not met by user "root"
Dec 01 06:48:14 --217-28.webhostbox.net sshd[20882]: Failed password for root from 183.201.252.68 port 56722 ssh2
Dec 01 06:48:14 --217-28.webhostbox.net sshd[20882]: Received disconnect from 183.201.252.68 port 56722:11: Bye Bye [preauth]
Dec 01 06:48:14 --217-28.webhostbox.net sshd[20882]: Disconnected from 183.201.252.68 port 56722 [preauth]
Dec 01 06:48:26 --217-28.webhostbox.net sshd[20977]: Invalid user ching from 114.174.161.11 port 61578
Dec 01 06:48:26 --217-28.webhostbox.net sshd[20977]: input_userauth_request: invalid user ching [preauth]
Dec 01 06:48:26 --217-28.webhostbox.net sshd[20977]: pam_unix(sshd:auth): check pass; user unknown
Dec 01 06:48:26 --217-28.webhostbox.net sshd[20977]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=114.174.161.11
Dec 01 06:48:28 --217-28.webhostbox.net sshd[20977]: Failed password for invalid user ching from 114.174.161.11 port 61578 ssh2
Dec 01 06:48:28 --217-28.webhostbox.net sshd[20977]: Received disconnect from 114.174.161.11 port 61578:11: Bye Bye [preauth]
Dec 01 06:48:28 --217-28.webhostbox.net sshd[20977]: Disconnected from 114.174.161.11 port 61578 [preauth]
lines 1984-2006/2006 (END)

search%20issue|690x181

[balu_thomas]

Hi Guys,

We tried with Elasticsearch 7.3.3 with SE JDK 11 or higher. We are facing this issue. But for lower version like Elasticsearch 1.7.3 no errors, we were able to start the service. The weird thing is that the same Elasticsearch 7.3.3 with SE JDK 11 is working fine on our dedicated server. VPS servers have this issue. Tried with two separate hosting providers. Please help us to find a root cause for this issue.

Thanks in Advance!

[spinscale (Alexander Reelsen)]

can you share some more dedicated elastic error message, either by using journalctl or check the logfiles in /var/log/elasticsearch, why this has failed?

See https://www.elastic.co/guide/en/elasticsearch/reference/7.4/starting-elasticsearch.html

--Alex

[balu_thomas]

@spinscale

/var/log/elasticsearch

Hi Alexandar,

Thanks for your kind help. We were able to fix the issue. We tried to get the elasticsearch.log file from the mentioned location(/var/log/elasticsearch). We found a critical error :
"unable to load JNA native support library, native methods will be disabled.
java.lang.UnsatisfiedLinkError: Native library (com/sun/jna/linux-x86/libjnidispatch.so) not found in resource path ([file:/usr/share/elasticsearch/lib/lucene-spatial-$"

Fix Applied:

1. Go to /etc/sysconfig/elasticsearch
2. Add ES_JAVA_OPTS="-Djna.tmpdir=/var/lib/elasticsearch/tmp"
3. start elasticsearch.
4. U can see now tmp folder created inside /var/lib/elasticsearch/.
5. This folder should have execute permission for "elasticsearch" user
6. check the permission with namei -l /var/lib/elasticsearch
7. dr-xr-xr-x root root / drwxr-xr-x root root var drwxr-xr-x root root lib drwxr-x--- elasticsearch elasticsearch elasticsearch.