Elasticsearch.service failed after restart

dirtdart666 · August 17, 2023, 8:31pm

Hi everyone,

I am attempitng to install Elasticsearch on an Ubuntu server 22.04. LTS VM but I am running into a few issues.

After configuring /etc/elasticsearch/elasticsearch.yml saving and then running sudo systemctl restart elasticsearch.service

I am hit with this error

Job for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xeu elasticsearch.service" for details.

Error output for systemctl status elasticsearch.service

Job for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xeu elasticsearch.service" for details.
elk@elastic-test1:~$ systemctl status elasticsearch.service
× elasticsearch.service - Elasticsearch
     Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2023-08-17 20:09:20 UTC; 50s ago
       Docs: https://www.elastic.co
    Process: 1351 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
   Main PID: 1351 (code=exited, status=1/FAILURE)
        CPU: 2.766s

 elastic-test1 systemd-entrypoint[1351]:         at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:238)
 elastic-test1 systemd-entrypoint[1351]:         at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:639)
 elastic-test1 systemd-entrypoint[1351]:         at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:162)
 elastic-test1 systemd-entrypoint[1351]:         at org.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:172)
 elastic-test1 systemd-entrypoint[1351]:         at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:429)
 elastic-test1 systemd-entrypoint[1351]:         ... 12 more
 elastic-test1 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
 elastic-test1 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
 elastic-test1 systemd[1]: Failed to start Elasticsearch.
 elastic-test1 systemd[1]: elasticsearch.service: Consumed 2.766s CPU time.

error output for journalctl -xeu elasticsearch.service

Job for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xeu elasticsearch.service" for details.
elk@elastic-test1:~$ systemctl status elasticsearch.service
× elasticsearch.service - Elasticsearch
     Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2023-08-17 20:09:20 UTC; 50s ago
       Docs: https://www.elastic.co
    Process: 1351 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
   Main PID: 1351 (code=exited, status=1/FAILURE)
        CPU: 2.766s

 elastic-test1 systemd-entrypoint[1351]:         at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:238)
 elastic-test1 systemd-entrypoint[1351]:         at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:639)
 elastic-test1 systemd-entrypoint[1351]:         at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:162)
 elastic-test1 systemd-entrypoint[1351]:         at org.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:172)
 elastic-test1 systemd-entrypoint[1351]:         at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:429)
 elastic-test1 systemd-entrypoint[1351]:         ... 12 more
 elastic-test1 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
 elastic-test1 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
 elastic-test1 systemd[1]: Failed to start Elasticsearch.
 elastic-test1 systemd[1]: elasticsearch.service: Consumed 2.766s CPU time.
 elastic-test1 systemd-entrypoint[1351]: found character '@' that cannot start any token. (Do not use @ for indentation)
 elastic-test1 systemd-entrypoint[1351]:  in 'reader', line 32, column 1:
 elastic-test1 systemd-entrypoint[1351]:     @path.data@
 elastic-test1 systemd-entrypoint[1351]:     ^
 elastic-test1 systemd-entrypoint[1351]:  at [Source: (ByteArrayInputStream); line: 23, column: 25]
 elastic-test1 systemd-entrypoint[1351]:         at org.elasticsearch.xcontent.provider.json.JsonXContentParser.newXContentParseException(JsonXContentParser.java:52)
 elastic-test1 systemd-entrypoint[1351]:         at org.elasticsearch.xcontent.provider.json.JsonXContentParser.nextFieldName(JsonXContentParser.java:71)
 elastic-test1 systemd-entrypoint[1351]:         at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:759)
 elastic-test1 systemd-entrypoint[1351]:         at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:727)
 elastic-test1 systemd-entrypoint[1351]:         at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1229)
 elastic-test1 systemd-entrypoint[1351]:         ... 7 more
 elastic-test1 systemd-entrypoint[1351]: Caused by: com.fasterxml.jackson.dataformat.yaml.snakeyaml.error.MarkedYAMLException: while scanning for the next token
 elastic-test1 systemd-entrypoint[1351]: found character '@' that cannot start any token. (Do not use @ for indentation)
 elastic-test1 systemd-entrypoint[1351]:  in 'reader', line 32, column 1:
 elastic-test1 systemd-entrypoint[1351]:     @path.data@
 elastic-test1 systemd-entrypoint[1351]:     ^
 elastic-test1 systemd-entrypoint[1351]:  at [Source: (ByteArrayInputStream); line: 23, column: 25]
 elastic-test1 systemd-entrypoint[1351]:         at com.fasterxml.jackson.dataformat.yaml.snakeyaml.error.MarkedYAMLException.from(MarkedYAMLException.java:28)
 elastic-test1 systemd-entrypoint[1351]:         at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:433)
 elastic-test1 systemd-entrypoint[1351]:         at com.fasterxml.jackson.core.JsonParser.nextFieldName(JsonParser.java:1091)
 elastic-test1 systemd-entrypoint[1351]:         at org.elasticsearch.xcontent.provider.json.JsonXContentParser.nextFieldName(JsonXContentParser.java:69)
 elastic-test1 systemd-entrypoint[1351]:         ... 10 more
 elastic-test1 systemd-entrypoint[1351]: Caused by: while scanning for the next token
 elastic-test1 systemd-entrypoint[1351]: found character '@' that cannot start any token. (Do not use @ for indentation)
 elastic-test1 systemd-entrypoint[1351]:  in 'reader', line 32, column 1:
 elastic-test1 systemd-entrypoint[1351]:     @path.data@
 elastic-test1 systemd-entrypoint[1351]:     ^
 elastic-test1 systemd-entrypoint[1351]:         at org.yaml.snakeyaml.scanner.ScannerImpl.fetchMoreTokens(ScannerImpl.java:445)
 elastic-test1 systemd-entrypoint[1351]:         at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:238)
 elastic-test1 systemd-entrypoint[1351]:         at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:639)
 elastic-test1 systemd-entrypoint[1351]:         at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:162)
 elastic-test1 systemd-entrypoint[1351]:         at org.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:172)
 elastic-test1 systemd-entrypoint[1351]:         at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:429)
 elastic-test1 systemd-entrypoint[1351]:         ... 12 more
 elastic-test1 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ An ExecStart= process belonging to unit elasticsearch.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
 elastic-test1 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit elasticsearch.service has entered the 'failed' state with result 'exit-code'.
 elastic-test1 systemd[1]: Failed to start Elasticsearch.
░░ Subject: A start job for unit elasticsearch.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit elasticsearch.service has finished with a failure.
░░
░░ The job identifier is 693 and the job result is failed.
 elastic-test1 systemd[1]: elasticsearch.service: Consumed 2.766s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit elasticsearch.service completed and consumed the indicated resources.

I ran journalctl -u elasticsearch.service and got the below out put

systemd[1]: Starting Elasticsearch...
systemd[1]: Started Elasticsearch.
systemd[1]: Stopping Elasticsearch...
systemd[1]: elasticsearch.service: Deactivated successfully.
systemd[1]: Stopped Elasticsearch.
systemd[1]: elasticsearch.service: Consumed 1min 5.740s CPU time.
systemd[1]: Starting Elasticsearch...
systemd-entrypoint[4036]: Exception in thread "main" org.elasticsearch.common.settings.SettingsException: Failed to load settings from [elasticsearch.yml]
systemd-entrypoint[4036]:         at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1233)
systemd-entrypoint[4036]:         at org.elasticsearch.node.InternalSettingsPreparer.loadConfigWithSubstitutions(InternalSettingsPreparer.java:126)
systemd-entrypoint[4036]:         at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:52)
systemd-entrypoint[4036]:         at org.elasticsearch.common.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:110)
systemd-entrypoint[4036]:         at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54)
systemd-entrypoint[4036]:         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85)
systemd-entrypoint[4036]:         at org.elasticsearch.cli.Command.main(Command.java:50)
systemd-entrypoint[4036]:         at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:64)
systemd-entrypoint[4036]: Caused by: org.elasticsearch.xcontent.XContentParseException: [109:29] Duplicate field 'cluster.initial_master_nodes'
systemd-entrypoint[4036]:  at [Source: (ByteArrayInputStream); line: 109, column: 29]
systemd-entrypoint[4036]:         at org.elasticsearch.xcontent.provider.json.JsonXContentParser.newXContentParseException(JsonXContentParser.java:52)
systemd-entrypoint[4036]:         at org.elasticsearch.xcontent.provider.json.JsonXContentParser.nextFieldName(JsonXContentParser.java:71)
systemd-entrypoint[4036]:         at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:759)
systemd-entrypoint[4036]:         at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:727)
systemd-entrypoint[4036]:         at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1229)
systemd-entrypoint[4036]:         ... 7 more
systemd-entrypoint[4036]: Caused by: com.fasterxml.jackson.core.JsonParseException: Duplicate field 'cluster.initial_master_nodes'
systemd-entrypoint[4036]:  at [Source: (ByteArrayInputStream); line: 109, column: 29]
systemd-entrypoint[4036]:         at com.fasterxml.jackson.core.json.JsonReadContext._checkDup(JsonReadContext.java:243)
systemd-entrypoint[4036]:         at com.fasterxml.jackson.core.json.JsonReadContext.setCurrentName(JsonReadContext.java:237)
systemd-entrypoint[4036]:         at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:482)
systemd-entrypoint[4036]:         at com.fasterxml.jackson.core.JsonParser.nextFieldName(JsonParser.java:1091)
systemd-entrypoint[4036]:         at org.elasticsearch.xcontent.provider.json.JsonXContentParser.nextFieldName(JsonXContentParser.java:69)
systemd-entrypoint[4036]:         ... 10 more

I noticed the following

Exception in thread "main" org.elasticsearch.common.settings.SettingsException: Failed to load settings from [elasticsearch.yml]

and

Caused by: org.elasticsearch.xcontent.XContentParseException: [109:29] Duplicate field 'cluster.initial_master_nodes'

There is only one 'cluster.initial_master_nodes' field in my elasticsearch.yml. I commented it out saved and restarted elasticsearch.service but Is still get the same error.

I have looked everywhere I can think of but I am out of ideas. I might even be on the wrong track...

I have had a look on the forums here and elsewhere but can't seem to find anything that anything that helped.

Can anyone help?

Thank you

DavidTurner · August 18, 2023, 6:16am

I think you're on the right track, Duplicate field 'cluster.initial_master_nodes' would definitely prevent Elasticsearch from starting. Note these docs indicating you shouldn't have it set at all:

IMPORTANT: After the cluster has formed, remove the cluster.initial_master_nodes setting from each node’s configuration. It should not be set for master-ineligible nodes, master-eligible nodes joining an existing cluster, or nodes which are restarting.

You can't get this error without genuinely having a duplicate cluster.initial_master_nodes setting. Perhaps you are not editing the config file that Elasticsearch is reading?

dirtdart666 · August 19, 2023, 3:01am

Hi David,

Thank you for the help and the link to the docs regarding cluster.initial_master_nodes.

I was editing /etc/elasticsearch/elasticsearch.yml That's the main config for Elasticsearch right?

I commented out cluster.initial_master_nodes configuration before but it didn't work.

I have obviously done something wrong somewhere...

I ended up rolling back the VM to a previous snapshot and starting again. This time I didn't add any cluster.initial_master_nodes`.' configuration and it works now.

Once again thanks for your help.

system · September 16, 2023, 3:02am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Failed to start Elasticsearch 7.11.2 Elasticsearch	36	1265	July 6, 2023
Can't start elasticsearch service Elasticsearch	9	1500	October 1, 2022
Error when start the elastics services Elasticsearch elastic-stack-monitoring , elastic-stack-security , elastic-stack-alerting , runtime-fields	2	677	January 16, 2024
Elasticsearch installation issues Elasticsearch	3	359	December 6, 2023
Main process exited, code=exited, status=1/FAILURE Elasticsearch	12	5366	March 22, 2022

Elasticsearch.service failed after restart

Related Topics