Elasticsearch not starting, with no errors in the logs

sphawk · January 7, 2023, 10:54am

I've updated debian to buster version without no problem and today I've tried to update ES to 8.5 version.
Bad idea (some Java problem) so I've reinstalled previous version (6.x) but ES still not starting.

service elasticsearch status return

● elasticsearch.service - Elasticsearch
   Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; disabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sat 2023-01-07 11:51:21 CET; 2s ago
     Docs: http://www.elastic.co
  Process: 710 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
 Main PID: 710 (code=exited, status=1/FAILURE)

Jan 07 11:51:21 pan elasticsearch[710]:         at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
Jan 07 11:51:21 pan elasticsearch[710]:         at java.base/java.nio.file.Files.newByteChannel(Files.java:371)
Jan 07 11:51:21 pan elasticsearch[710]:         at java.base/java.nio.file.Files.newByteChannel(Files.java:422)
Jan 07 11:51:21 pan elasticsearch[710]:         at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420)
Jan 07 11:51:21 pan elasticsearch[710]:         at java.base/java.nio.file.Files.newInputStream(Files.java:156)
Jan 07 11:51:21 pan elasticsearch[710]:         at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1112)
Jan 07 11:51:21 pan elasticsearch[710]:         at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:100)
Jan 07 11:51:21 pan elasticsearch[710]:         ... 6 more
Jan 07 11:51:21 pan systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Jan 07 11:51:21 pan systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

and /var/log/elasticsearch contains only gc.log

[2023-01-07T10:51:20.463+0000][710][gc] Using Concurrent Mark Sweep
[2023-01-07T10:51:20.463+0000][710][gc,heap,coops] Heap address: 0x00000000c0000000, size: 1024 MB, Compressed Oops mode: 32-bit
[2023-01-07T10:51:20.520+0000][710][safepoint    ] Entering safepoint region: EnableBiasedLocking
[2023-01-07T10:51:20.520+0000][710][safepoint    ] Leaving safepoint region
[2023-01-07T10:51:20.520+0000][710][safepoint    ] Total time for which application threads were stopped: 0,0001924 seconds, Stopping threads took: 0,0000840 seconds
[2023-01-07T10:51:20.525+0000][710][safepoint    ] Application time: 0,0038243 seconds
[2023-01-07T10:51:20.525+0000][710][safepoint    ] Entering safepoint region: RevokeBias
[2023-01-07T10:51:20.525+0000][710][safepoint    ] Leaving safepoint region
[2023-01-07T10:51:20.525+0000][710][safepoint    ] Total time for which application threads were stopped: 0,0001419 seconds, Stopping threads took: 0,0000292 seconds
[2023-01-07T10:51:20.642+0000][710][safepoint    ] Application time: 0,1168887 seconds
[2023-01-07T10:51:20.642+0000][710][safepoint    ] Entering safepoint region: RevokeBias
[2023-01-07T10:51:20.642+0000][710][safepoint    ] Leaving safepoint region
[2023-01-07T10:51:20.642+0000][710][safepoint    ] Total time for which application threads were stopped: 0,0001366 seconds, Stopping threads took: 0,0000179 seconds
[2023-01-07T10:51:20.644+0000][710][safepoint    ] Application time: 0,0019418 seconds
[2023-01-07T10:51:20.644+0000][710][safepoint    ] Entering safepoint region: Deoptimize
[2023-01-07T10:51:20.644+0000][710][safepoint    ] Leaving safepoint region
[2023-01-07T10:51:20.644+0000][710][safepoint    ] Total time for which application threads were stopped: 0,0000857 seconds, Stopping threads took: 0,0000140 seconds
[2023-01-07T10:51:20.674+0000][710][safepoint    ] Application time: 0,0299137 seconds
[2023-01-07T10:51:20.674+0000][710][safepoint    ] Entering safepoint region: Deoptimize
[2023-01-07T10:51:20.674+0000][710][safepoint    ] Leaving safepoint region
[2023-01-07T10:51:20.674+0000][710][safepoint    ] Total time for which application threads were stopped: 0,0001231 seconds, Stopping threads took: 0,0000190 seconds
[2023-01-07T10:51:20.674+0000][710][safepoint    ] Application time: 0,0003416 seconds
[2023-01-07T10:51:20.674+0000][710][safepoint    ] Entering safepoint region: Deoptimize
[2023-01-07T10:51:20.674+0000][710][safepoint    ] Leaving safepoint region
[2023-01-07T10:51:20.674+0000][710][safepoint    ] Total time for which application threads were stopped: 0,0000517 seconds, Stopping threads took: 0,0000137 seconds
[2023-01-07T10:51:21.098+0000][710][safepoint    ] Application time: 0,4236951 seconds
[2023-01-07T10:51:21.098+0000][710][safepoint    ] Entering safepoint region: Deoptimize
[2023-01-07T10:51:21.098+0000][710][safepoint    ] Leaving safepoint region
[2023-01-07T10:51:21.098+0000][710][safepoint    ] Total time for which application threads were stopped: 0,0002734 seconds, Stopping threads took: 0,0000198 seconds
[2023-01-07T10:51:21.137+0000][710][safepoint    ] Application time: 0,0387620 seconds
[2023-01-07T10:51:21.137+0000][710][safepoint    ] Entering safepoint region: Deoptimize
[2023-01-07T10:51:21.137+0000][710][safepoint    ] Leaving safepoint region
[2023-01-07T10:51:21.137+0000][710][safepoint    ] Total time for which application threads were stopped: 0,0002025 seconds, Stopping threads took: 0,0000187 seconds
[2023-01-07T10:51:21.287+0000][710][safepoint    ] Application time: 0,1496150 seconds
[2023-01-07T10:51:21.287+0000][710][safepoint    ] Entering safepoint region: Deoptimize
[2023-01-07T10:51:21.287+0000][710][safepoint    ] Leaving safepoint region
[2023-01-07T10:51:21.287+0000][710][safepoint    ] Total time for which application threads were stopped: 0,0002512 seconds, Stopping threads took: 0,0000192 seconds
[2023-01-07T10:51:21.314+0000][710][safepoint    ] Application time: 0,0263280 seconds
[2023-01-07T10:51:21.314+0000][710][safepoint    ] Entering safepoint region: RevokeBias
[2023-01-07T10:51:21.314+0000][710][safepoint    ] Leaving safepoint region
[2023-01-07T10:51:21.314+0000][710][safepoint    ] Total time for which application threads were stopped: 0,0002161 seconds, Stopping threads took: 0,0001115 seconds
[2023-01-07T10:51:21.366+0000][710][gc,heap,exit ] Heap
[2023-01-07T10:51:21.366+0000][710][gc,heap,exit ]  par new generation   total 314560K, used 61557K [0x00000000c0000000, 0x00000000d5550000, 0x00000000d5550000)
[2023-01-07T10:51:21.366+0000][710][gc,heap,exit ]   eden space 279616K,  22% used [0x00000000c0000000, 0x00000000c3c1d558, 0x00000000d1110000)
[2023-01-07T10:51:21.366+0000][710][gc,heap,exit ]   from space 34944K,   0% used [0x00000000d1110000, 0x00000000d1110000, 0x00000000d3330000)
[2023-01-07T10:51:21.366+0000][710][gc,heap,exit ]   to   space 34944K,   0% used [0x00000000d3330000, 0x00000000d3330000, 0x00000000d5550000)
[2023-01-07T10:51:21.366+0000][710][gc,heap,exit ]  concurrent mark-sweep generation total 699072K, used 0K [0x00000000d5550000, 0x0000000100000000, 0x0000000100000000)
[2023-01-07T10:51:21.366+0000][710][gc,heap,exit ]  Metaspace       used 9463K, capacity 10904K, committed 11008K, reserved 1058816K
[2023-01-07T10:51:21.366+0000][710][gc,heap,exit ]   class space    used 1254K, capacity 1513K, committed 1536K, reserved 1048576K
[2023-01-07T10:51:21.366+0000][710][safepoint    ] Application time: 0,0522783 seconds
[2023-01-07T10:51:21.366+0000][710][safepoint    ] Entering safepoint region: Halt

any idea?
I hope I don't have to reinstall everything...

sphawk · January 7, 2023, 11:18am

on /var/log/syslog

Jan  7 12:14:09 pan systemd[1]: Started Elasticsearch.
Jan  7 12:14:09 pan elasticsearch[31281]: warning: Falling back to java on path. This behavior is deprecated. Specify JAVA_HOME
Jan  7 12:14:10 pan elasticsearch[31281]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Jan  7 12:14:10 pan elasticsearch[31281]: OpenJDK 64-Bit Server VM warning: UseAVX=2 is not supported on this CPU, setting it to UseAVX=0
Jan  7 12:14:12 pan elasticsearch[31281]: Exception in thread "main" SettingsException[Failed to load settings from /etc/elasticsearch/elasticsearch.yml]; nested: AccessDeniedException[/etc/elasticsearch/elasticsearch.yml];
Jan  7 12:14:12 pan elasticsearch[31281]: #011at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:102)
Jan  7 12:14:12 pan elasticsearch[31281]: #011at org.elasticsearch.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:95)
Jan  7 12:14:12 pan elasticsearch[31281]: #011at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
Jan  7 12:14:12 pan elasticsearch[31281]: #011at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124)
Jan  7 12:14:12 pan elasticsearch[31281]: #011at org.elasticsearch.cli.Command.main(Command.java:90)
Jan  7 12:14:12 pan elasticsearch[31281]: #011at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:116)
Jan  7 12:14:12 pan elasticsearch[31281]: #011at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:93)
Jan  7 12:14:12 pan elasticsearch[31281]: Caused by: java.nio.file.AccessDeniedException: /etc/elasticsearch/elasticsearch.yml
Jan  7 12:14:12 pan elasticsearch[31281]: #011at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
Jan  7 12:14:12 pan elasticsearch[31281]: #011at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
Jan  7 12:14:12 pan elasticsearch[31281]: #011at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
Jan  7 12:14:12 pan elasticsearch[31281]: #011at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
Jan  7 12:14:12 pan elasticsearch[31281]: #011at java.base/java.nio.file.Files.newByteChannel(Files.java:371)
Jan  7 12:14:12 pan elasticsearch[31281]: #011at java.base/java.nio.file.Files.newByteChannel(Files.java:422)
Jan  7 12:14:12 pan elasticsearch[31281]: #011at java.base/java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:420)
Jan  7 12:14:12 pan elasticsearch[31281]: #011at java.base/java.nio.file.Files.newInputStream(Files.java:156)
Jan  7 12:14:12 pan elasticsearch[31281]: #011at org.elasticsearch.common.settings.Settings$Builder.loadFromPath(Settings.java:1112)
Jan  7 12:14:12 pan elasticsearch[31281]: #011at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:100)
Jan  7 12:14:12 pan elasticsearch[31281]: #011... 6 more
Jan  7 12:14:12 pan systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
Jan  7 12:14:12 pan systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

Christian_Dahlqvist · January 7, 2023, 11:26am

Jan 7 12:14:12 pan elasticsearch[31281]: Exception in thread "main" SettingsException[Failed to load settings from /etc/elasticsearch/elasticsearch.yml]; nested: AccessDeniedException[/etc/elasticsearch/elasticsearch.yml];

Did you look at the logs yourself? This is in the logs you shared and outlines the issue quite clearly IMHO.

sphawk · January 7, 2023, 11:29am

the problem was the symlink
fixed with

rm elasticsearch.yml && mv elasticsearch.yml.archimed elasticsearch.yml

sorry

system · February 4, 2023, 11:29am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.