What does this error mean, and would it cause a data node to crash

Elastic Stack Elasticsearch
1

I have a cluster on ES 5.x and suddenly some of the data nodes simply crash randomly. I see this error and want to know what it is and is this something I'm overlooking to fix it??? Any insight would be GREATLY appreciated.

elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: failed (Result: exit-code) since Tue 2021-02-02 17:41:29 EST; 15h ago
Docs: http://www.elastic.co
Process: 1108 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p {PID_DIR}/elasticsearch.pid --quiet -Edefault.path.logs={LOG_D
Process: 1069 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 1108 (code=exited, status=127)

Feb 02 11:26:57 ITSSD5ESDN04 elasticsearch[1108]: 2021-02-02 11:26:57,863 main ERROR Null object returned for RollingFile in Appenders.
Feb 02 11:26:57 ITSSD5ESDN04 elasticsearch[1108]: 2021-02-02 11:26:57,863 main ERROR Null object returned for RollingFile in Appenders.
Feb 02 11:26:57 ITSSD5ESDN04 elasticsearch[1108]: 2021-02-02 11:26:57,864 main ERROR Null object returned for RollingFile in Appenders.
Feb 02 11:26:57 ITSSD5ESDN04 elasticsearch[1108]: 2021-02-02 11:26:57,864 main ERROR Unable to locate appender "rolling" for logger conf
Feb 02 11:26:57 ITSSD5ESDN04 elasticsearch[1108]: 2021-02-02 11:26:57,864 main ERROR Unable to locate appender "index_indexing_slowlog_r
Feb 02 11:26:57 ITSSD5ESDN04 elasticsearch[1108]: 2021-02-02 11:26:57,864 main ERROR Unable to locate appender "index_search_slowlog_rol
Feb 02 11:26:57 ITSSD5ESDN04 elasticsearch[1108]: 2021-02-02 11:26:57,864 main ERROR Unable to locate appender "deprecation_rolling" for
Feb 02 17:41:29 ITSSD5ESDN04 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=127/n/a
Feb 02 17:41:29 ITSSD5ESDN04 systemd[1]: elasticsearch.service: Unit entered failed state.
Feb 02 17:41:29 ITSSD5ESDN04 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.

2

That's EOL, you should upgrade as a matter of urgency. Latest versions are 6.8 and 7.10.

Is there more in your Elasticsearch log file?

3

Yeah, we plan to, but i need to fix this now. Just trying to figure out whats causing the Datanode crashing now so i can upgrade.

4

Also,
Where exactly would that log file reside?
Thanks

5

Depends how you installed it, but usually /var/log/elasticsearch.

6

Good Day,

I check that directory and I'm not seeing any files / log files at all there, is there a place I can check to make sure logging is running and to where?

7

I think i found the directory i have my log files in the elastic yml file. But im not seeing any log files there either. So do i need to do something to set the permissions to the custom directory?

also,
I have my data nodes with 64gb of ram and i have the jvm memory set for a little than less than half of that... 29g.
Im experiencing data node crashes, should set / uncomment the config line... bootstrap.memory_lock: true

thanks,

8

If Elasticsearch cannot log then it won't start. There's nothing in that specified log path?

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.