Unit elasticsearch.service has failed to start on centos 7

dan_bau · November 25, 2021, 10:43pm

Can you please help?

rpm -qa|grep elasticsearch
elasticsearch-7.15.2-1.x86_64

Nov 25 23:20:01 vps.ovh.net crond[2260]: WARNING: RunAsUser for MSP ignored, check group ids (egid=48, want=51)
Nov 25 23:20:01 vps.ovh.net crond[2260]: Program mode requires special privileges, e.g., root or TrustedUser.
Nov 25 23:20:01 vps.ovh.net CROND[14236]: (apache) MAIL (mailed 92 bytes of output but got status 0x004e
                                                       )
Nov 25 23:20:02 vps.ovh.net systemd-entrypoint[14098]: Exception in thread "main" java.nio.file.AccessDeniedException: /etc/elasticsearch/elasticsearch.keystore
Nov 25 23:20:02 vps.ovh.net systemd-entrypoint[14098]: at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
Nov 25 23:20:02 vps.ovh.net systemd-entrypoint[14098]: at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
Nov 25 23:20:02 .vps.ovh.net systemd-entrypoint[14098]: at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
Nov 25 23:20:02 vps.ovh.net systemd-entrypoint[14098]: at java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:182)
Nov 25 23:20:02 .vps.ovh.net systemd-entrypoint[14098]: at java.base/java.nio.channels.FileChannel.open(FileChannel.java:298)
Nov 25 23:20:02 vps.ovh.net systemd-entrypoint[14098]: at java.base/java.nio.channels.FileChannel.open(FileChannel.java:357)
Nov 25 23:20:02 vps.ovh.net systemd-entrypoint[14098]: at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:83)
Nov 25 23:20:02 vps.ovh.net systemd-entrypoint[14098]: at org.elasticsearch.common.settings.KeyStoreWrapper.load(KeyStoreWrapper.java:210)
Nov 25 23:20:02 vps.ovh.net systemd-entrypoint[14098]: at org.elasticsearch.common.settings.HasPasswordKeyStoreCommand.execute(HasPasswordKeyStoreCommand.java:32)
Nov 25 23:20:02 ]vps.ovh.net systemd-entrypoint[14098]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
Nov 25 23:20:02 .vps.ovh.net systemd-entrypoint[14098]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:114)
Nov 25 23:20:02 vps.ovh.net systemd-entrypoint[14098]: at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:95)
Nov 25 23:20:02 vps.ovh.net systemd-entrypoint[14098]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:114)
Nov 25 23:20:02 vps.ovh.net systemd-entrypoint[14098]: at org.elasticsearch.cli.Command.main(Command.java:79)
Nov 25 23:20:02 vps.ovh.net systemd-entrypoint[14098]: at org.elasticsearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:32)
Nov 25 23:20:06 vps.ovh.net systemd-entrypoint[14098]: WARNING: A terminally deprecated method in java.lang.System has been called
Nov 25 23:20:06 vps.ovh.net systemd-entrypoint[14098]: WARNING: System::setSecurityManager has been called by org.elasticsearch.bootstrap.Elasticsearch (file:/usr/share/elasticsearch/lib/elasticsearch-7.15.2.jar)
Nov 25 23:20:06 vps.ovh.net systemd-entrypoint[14098]: WARNING: Please consider reporting this to the maintainers of org.elasticsearch.bootstrap.Elasticsearch
Nov 25 23:20:06 vps.ovh.net systemd-entrypoint[14098]: WARNING: System::setSecurityManager will be removed in a future release
Nov 25 23:20:06 vps.ovh.net systemd-entrypoint[14098]: Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.nio.file.AccessDeniedException: /etc/elasticsearch/elasticsearch.keystore
Nov 25 23:20:06 vps.ovh.net systemd-entrypoint[14098]: Likely root cause: java.nio.file.AccessDeniedException: /etc/elasticsearch/elasticsearch.keystore
Nov 25 23:20:06 vps.ovh.net systemd-entrypoint[14098]: at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
Nov 25 23:20:06 vps.ovh.net systemd-entrypoint[14098]: at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
Nov 25 23:20:06 vps.ovh.net systemd-entrypoint[14098]: at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
Nov 25 23:20:06 vps.ovh.net systemd-entrypoint[14098]: at java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:182)
Nov 25 23:20:06 vps.ovh.net systemd-entrypoint[14098]: at java.base/java.nio.channels.FileChannel.open(FileChannel.java:298)
Nov 25 23:20:06 vps.ovh.net systemd-entrypoint[14098]: at java.base/java.nio.channels.FileChannel.open(FileChannel.java:357)
Nov 25 23:20:06 vps.ovh.net systemd-entrypoint[14098]: at org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:83)
Nov 25 23:20:06 vps.ovh.net systemd-entrypoint[14098]: at org.elasticsearch.common.settings.KeyStoreWrapper.load(KeyStoreWrapper.java:210)
Nov 25 23:20:06 vps.ovh.net systemd-entrypoint[14098]: at org.elasticsearch.bootstrap.Bootstrap.loadSecureSettings(Bootstrap.java:236)
Nov 25 23:20:06 vps.ovh.net systemd-entrypoint[14098]: at org.elasticsearch.bootstrap.Bootstrap.loadSecureSettings(Bootstrap.java:230)
Nov 25 23:20:06 vps.ovh.net systemd-entrypoint[14098]: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:345)
Nov 25 23:20:06 vps.ovh.net systemd-entrypoint[14098]: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:167)
Nov 25 23:20:06 vps.ovh.net systemd-entrypoint[14098]: at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:158)
Nov 25 23:20:06 vps.ovh.net systemd-entrypoint[14098]: at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
Nov 25 23:20:06 vps.ovh.net systemd-entrypoint[14098]: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:114)
Nov 25 23:20:06 vps.ovh.net systemd-entrypoint[14098]: at org.elasticsearch.cli.Command.main(Command.java:79)
Nov 25 23:20:06 vps.ovh.net systemd-entrypoint[14098]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:123)
Nov 25 23:20:06 vps.ovh.net systemd-entrypoint[14098]: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:81)
Nov 25 23:20:07 vps.ovh.net systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 25 23:20:07 vps.ovh.net systemd[1]: Failed to start Elasticsearch.
-- Subject: Unit elasticsearch.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has failed.
--
-- The result is failed.
Nov 25 23:20:07 vps.ovh.net systemd[1]: Unit elasticsearch.service entered failed state.
Nov 25 23:20:07 vps.ovh.net systemd[1]: elasticsearch.service failed.
Nov 25 23:20:07 vps.ovh.net polkitd[601]: Unregistered Authentication Agent for unix-process:14092:608728813 (system bus name :1.849772, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disco
l

can you please help ?

warkolm · November 25, 2021, 10:56pm

Welcome to our community!

Can you check that?

dan_bau · November 30, 2021, 12:02am

oh I did it. Not too much luck
it looks like

?×lelasticsearch.keystore “ @v!'‡ói5´I;G%ý ©—g›kÚ{õî¿ñdh6ÖäEgÜ¡T‹•júH{I–°‚úHÿæÐéUk…4m +s|ç|Ÿî€fia+ ;Põt<“»›l¢¿”LDtë-Ç„§n¨~ÖÙùxée,5,LýbÌ[U M8ó–r-4R˜'ãCŠÀ(“è sÊc'`

dan_bau · November 30, 2021, 12:15am

Exception in thread "main" org.apache.lucene.index.CorruptIndexException: codec header mismatch: actual header=1069782892 vs expected header=1071082519 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/etc/elasticsearch/elasticsearch.keystore")))
at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:196)
at org.elasticsearch.common.settings.KeyStoreWrapper.load(KeyStoreWrapper.java:214)
at org.elasticsearch.common.settings.HasPasswordKeyStoreCommand.execute(HasPasswordKeyStoreCommand.java:32)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:114)
at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:95)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:114)
at org.elasticsearch.cli.Command.main(Command.java:79)
at org.elasticsearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:32)
Exception in thread "main" org.apache.lucene.index.CorruptIndexException: codec header mismatch: actual header=1069782892 vs expected header=1071082519 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/etc/elasticsearch/elasticsearch.keystore")))
at org.apache.lucene.codecs.CodecUtil.checkHeader(CodecUtil.java:196)
at org.elasticsearch.common.settings.KeyStoreWrapper.load(KeyStoreWrapper.java:214)
at org.elasticsearch.common.settings.BaseKeyStoreCommand.execute(BaseKeyStoreCommand.java:39)
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:75)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:114)
at org.elasticsearch.cli.MultiCommand.execute(MultiCommand.java:95)
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:114)
at org.elasticsearch.cli.Command.main(Command.java:79)
at org.elasticsearch.common.settings.KeyStoreCli.main(KeyStoreCli.java:32)
warning: %posttrans(elasticsearch-0:7.15.2-1.x86_64) scriptlet failed, exit status 1
Non-fatal POSTTRANS scriptlet failure in rpm package elasticsearch-7.15.2-1.x86_64
Verifying : elasticsearch-7.15.2-1.x86_64Preformatted text

dan_bau · November 30, 2021, 5:01pm

any idea how to fix those errors. too


Nov 30 17:59:01 vps.ovh.net sendmail[15645]: NOQUEUE: SYSERR(apache): can not chdir(/var/spool/clientmqueue/): Permission denied
Nov 30 17:59:01 vps.ovh.net crond[2260]: Program mode requires special privileges, e.g., root or TrustedUser.
Nov 30 17:59:01 vps.ovh.net CROND[15629]: (apache) MAIL (mailed 92 bytes of output but got status 0x004e
                                                       )
Nov 30 17:59:12 vps.ovh.net saslauthd[617]: pam_unix(smtp:auth): check pass; user unknown
Nov 30 17:59:12 vps.ovh.net saslauthd[617]: pam_unix(smtp:auth): authentication failure; logname= uid=0 euid=0 tty= ruser= rhost=
Nov 30 17:59:14 vps.ovh.net saslauthd[617]: DEBUG: auth_pam: pam_authenticate failed: User not known to the underlying authentication module
Nov 30 17:59:14 vps.ovh.net saslauthd[617]: do_auth         : auth failure: [user=karol] [service=smtp] [realm=ovh.net] [mech=pam] [reason=PAM auth error]
Nov 30 17:59:14 vps.ovh.net sendmail[15621]: 1AUGwffG015621: AUTH failure (LOGIN): authentication failure (-13) SASL(-13): authentication failure: checkpass failed, rela failed, relay=[212.70.149.72]
Nov 30 17:59:18 vps.ovh.net sendmail[15621]: 1AUGwffG015621: [212.70.149.72] did not issue MAIL/EXPN/VRFY/ETRN during connection to TLSMTA

dan_bau · December 2, 2021, 12:26am

please HELP!!!!

[root@vps-5 magento]# service elasticsearch status -l
● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: disabled)
   Active: failed (Result: signal) since Wed 2021-12-01 23:14:06 CET; 2min 21s ago
     Docs: https://www.elastic.co
  Process: 6831 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=killed, signal=KILL)
 Main PID: 6831 (code=killed, signal=KILL)

Dec 01 23:13:28 vps.ovh.net systemd-entrypoint[6831]: WARNING: Please consider reporting this to the maintainers of org.elasticsearch.bootstrap.Elasticsearch
Dec 01 23:13:28 vps.ovh.net systemd-entrypoint[6831]: WARNING: System::setSecurityManager will be removed in a future release
Dec 01 23:13:30 vps.ovh.net systemd-entrypoint[6831]: WARNING: A terminally deprecated method in java.lang.System has been called
Dec 01 23:13:30 vps.ovh.net systemd-entrypoint[6831]: WARNING: System::setSecurityManager has been called by org.elasticsearch.bootstrap.Security (file:/usr/...15.2.jar)
Dec 01 23:13:30 vps.ovh.net systemd-entrypoint[6831]: WARNING: Please consider reporting this to the maintainers of org.elasticsearch.bootstrap.Security
Dec 01 23:13:30 vps.ovh.net systemd-entrypoint[6831]: WARNING: System::setSecurityManager will be removed in a future release
Dec 01 23:13:43 vps.ovh.net systemd[1]: Started Elasticsearch.
Dec 01 23:14:06 vps.ovh.net systemd[1]: elasticsearch.service: main process exited, code=killed, status=9/KILL
Dec 01 23:14:06 vps.ovh.net systemd[1]: Unit elasticsearch.service entered failed state.
Dec 01 23:14:06 vps.ovh.net systemd[1]: elasticsearch.service failed.
Hint: Some lines were ellipsized, use -l to show in full.




 journalctl -xe
-- Subject: Unit session-447019.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit session-447019.scope has finished starting up.
--
-- The start-up result is done.
Dec 02 01:05:01 vps-830fd688.vps.ovh.net systemd[1]: Started Session 447020 of user apache.
-- Subject: Unit session-447020.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit session-447020.scope has finished starting up.
--
-- The start-up result is done.
Dec 02 01:05:01 vps-830fd688.vps.ovh.net systemd[1]: Started Session 447021 of user apache.
-- Subject: Unit session-447021.scope has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit session-447021.scope has finished starting up.
Dec 02 01:05:01 vps.ovh.net CROND[18856]: (apache) CMD (/usr/bin/php /var/www/html/magentosite/bin/magento cron:run 2>&1 | grep -v "Ran jobs by schedule" >> /var/www/
Dec 02 01:05:01 vps.ovh.net crond[2260]: WARNING: RunAsUser for MSP ignored, check group ids (egid=48, want=51)
Dec 02 01:05:01 vps.ovh.net sendmail[18867]: NOQUEUE: SYSERR(apache): can not chdir(/var/spool/clientmqueue/): Permission denied
 Program mode requires special privileges, e.g., root or TrustedUser.
Dec 02 01:05:01 vps.ovh.net CROND[18851]: (apache) MAIL (mailed 92 bytes of output but got status 0x004e
                                                       )
Dec 02 01:05:08 vps.ovh.net clamd[8270]: SelfCheck: Database status OK.

warkolm · December 2, 2021, 1:20am

We need to see your Elasticsearch logs please, all of them.

dan_bau · December 2, 2021, 3:26am

Here they are

dan_bau · December 5, 2021, 2:09pm

can you help?
● Elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/Elasticsearch.service; disabled; vendor preset: disabled)
Active: failed (Result: signal) since Sun 2021-12-05 14:47:37 CET; 16min ago
Docs: https://www.elastic.co
Process: 2990 ExecStart=/usr/share/Elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/Elasticsearch.pid --quiet (code=killed, signal=KILL)
Main PID: 2990 (code=killed, signal=KILL)

dan_bau · December 5, 2021, 2:19pm

Dec 05 15:18:04 vps.ovh.net kernel: Out of memory: Kill process 6816 (java) score 537 or sacrifice child
Dec 05 15:18:04 vps.ovh.net kernel: Killed process 6816 (java), UID 986, total-vm:14232192kB, anon-rss:8516224kB, file-rss:2056kB, shmem-rss:0k

warkolm · December 5, 2021, 11:49pm

It's a lot easier if you can use gist/pastebin/etc for logs, instead of having people download them.

That last log would suggest you have an OOM. What is the output from the _cluster/stats?pretty&human API?

system · January 2, 2022, 11:49pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.