Elasticsearch.service failed after restart

dirtdart666 · August 17, 2023, 8:31pm

Hi everyone,

I am attempitng to install Elasticsearch on an Ubuntu server 22.04. LTS VM but I am running into a few issues.

After configuring /etc/elasticsearch/elasticsearch.yml saving and then running sudo systemctl restart elasticsearch.service

I am hit with this error

Job for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xeu elasticsearch.service" for details.

Error output for systemctl status elasticsearch.service

Job for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xeu elasticsearch.service" for details.
elk@elastic-test1:~$ systemctl status elasticsearch.service
× elasticsearch.service - Elasticsearch
     Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2023-08-17 20:09:20 UTC; 50s ago
       Docs: https://www.elastic.co
    Process: 1351 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
   Main PID: 1351 (code=exited, status=1/FAILURE)
        CPU: 2.766s

 elastic-test1 systemd-entrypoint[1351]:         at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:238)
 elastic-test1 systemd-entrypoint[1351]:         at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:639)
 elastic-test1 systemd-entrypoint[1351]:         at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:162)
 elastic-test1 systemd-entrypoint[1351]:         at org.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:172)
 elastic-test1 systemd-entrypoint[1351]:         at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:429)
 elastic-test1 systemd-entrypoint[1351]:         ... 12 more
 elastic-test1 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
 elastic-test1 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
 elastic-test1 systemd[1]: Failed to start Elasticsearch.
 elastic-test1 systemd[1]: elasticsearch.service: Consumed 2.766s CPU time.

error output for journalctl -xeu elasticsearch.service

Job for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xeu elasticsearch.service" for details.
elk@elastic-test1:~$ systemctl status elasticsearch.service
× elasticsearch.service - Elasticsearch
     Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2023-08-17 20:09:20 UTC; 50s ago
       Docs: https://www.elastic.co
    Process: 1351 ExecStart=/usr/share/elasticsearch/bin/systemd-entrypoint -p ${PID_DIR}/elasticsearch.pid --quiet (code=exited, status=1/FAILURE)
   Main PID: 1351 (code=exited, status=1/FAILURE)
        CPU: 2.766s

 elastic-test1 systemd-entrypoint[1351]:         at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:238)
 elastic-test1 systemd-entrypoint[1351]:         at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:639)
 elastic-test1 systemd-entrypoint[1351]:         at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:162)
 elastic-test1 systemd-entrypoint[1351]:         at org.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:172)
 elastic-test1 systemd-entrypoint[1351]:         at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:429)
 elastic-test1 systemd-entrypoint[1351]:         ... 12 more
 elastic-test1 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
 elastic-test1 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
 elastic-test1 systemd[1]: Failed to start Elasticsearch.
 elastic-test1 systemd[1]: elasticsearch.service: Consumed 2.766s CPU time.
 elastic-test1 systemd-entrypoint[1351]: found character '@' that cannot start any token. (Do not use @ for indentation)
 elastic-test1 systemd-entrypoint[1351]:  in 'reader', line 32, column 1:
 elastic-test1 systemd-entrypoint[1351]:     @path.data@
 elastic-test1 systemd-entrypoint[1351]:     ^
 elastic-test1 systemd-entrypoint[1351]:  at [Source: (ByteArrayInputStream); line: 23, column: 25]
 elastic-test1 systemd-entrypoint[1351]:         at org.elasticsearch.xcontent.provider.json.JsonXContentParser.newXContentParseException(JsonXContentParser.java:52)
 elastic-test1 systemd-entrypoint[1351]:         at org.elasticsearch.xcontent.provider.json.JsonXContentParser.nextFieldName(JsonXContentParser.java:71)
 elastic-test1 systemd-entrypoint[1351]:         at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:759)
 elastic-test1 systemd-entrypoint[1351]:         at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:727)
 elastic-test1 systemd-entrypoint[1351]:         at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1229)
 elastic-test1 systemd-entrypoint[1351]:         ... 7 more
 elastic-test1 systemd-entrypoint[1351]: Caused by: com.fasterxml.jackson.dataformat.yaml.snakeyaml.error.MarkedYAMLException: while scanning for the next token
 elastic-test1 systemd-entrypoint[1351]: found character '@' that cannot start any token. (Do not use @ for indentation)
 elastic-test1 systemd-entrypoint[1351]:  in 'reader', line 32, column 1:
 elastic-test1 systemd-entrypoint[1351]:     @path.data@
 elastic-test1 systemd-entrypoint[1351]:     ^
 elastic-test1 systemd-entrypoint[1351]:  at [Source: (ByteArrayInputStream); line: 23, column: 25]
 elastic-test1 systemd-entrypoint[1351]:         at com.fasterxml.jackson.dataformat.yaml.snakeyaml.error.MarkedYAMLException.from(MarkedYAMLException.java:28)
 elastic-test1 systemd-entrypoint[1351]:         at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:433)
 elastic-test1 systemd-entrypoint[1351]:         at com.fasterxml.jackson.core.JsonParser.nextFieldName(JsonParser.java:1091)
 elastic-test1 systemd-entrypoint[1351]:         at org.elasticsearch.xcontent.provider.json.JsonXContentParser.nextFieldName(JsonXContentParser.java:69)
 elastic-test1 systemd-entrypoint[1351]:         ... 10 more
 elastic-test1 systemd-entrypoint[1351]: Caused by: while scanning for the next token
 elastic-test1 systemd-entrypoint[1351]: found character '@' that cannot start any token. (Do not use @ for indentation)
 elastic-test1 systemd-entrypoint[1351]:  in 'reader', line 32, column 1:
 elastic-test1 systemd-entrypoint[1351]:     @path.data@
 elastic-test1 systemd-entrypoint[1351]:     ^
 elastic-test1 systemd-entrypoint[1351]:         at org.yaml.snakeyaml.scanner.ScannerImpl.fetchMoreTokens(ScannerImpl.java:445)
 elastic-test1 systemd-entrypoint[1351]:         at org.yaml.snakeyaml.scanner.ScannerImpl.checkToken(ScannerImpl.java:238)
 elastic-test1 systemd-entrypoint[1351]:         at org.yaml.snakeyaml.parser.ParserImpl$ParseBlockMappingKey.produce(ParserImpl.java:639)
 elastic-test1 systemd-entrypoint[1351]:         at org.yaml.snakeyaml.parser.ParserImpl.peekEvent(ParserImpl.java:162)
 elastic-test1 systemd-entrypoint[1351]:         at org.yaml.snakeyaml.parser.ParserImpl.getEvent(ParserImpl.java:172)
 elastic-test1 systemd-entrypoint[1351]:         at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:429)
 elastic-test1 systemd-entrypoint[1351]:         ... 12 more
 elastic-test1 systemd[1]: elasticsearch.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ An ExecStart= process belonging to unit elasticsearch.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 1.
 elastic-test1 systemd[1]: elasticsearch.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit elasticsearch.service has entered the 'failed' state with result 'exit-code'.
 elastic-test1 systemd[1]: Failed to start Elasticsearch.
░░ Subject: A start job for unit elasticsearch.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit elasticsearch.service has finished with a failure.
░░
░░ The job identifier is 693 and the job result is failed.
 elastic-test1 systemd[1]: elasticsearch.service: Consumed 2.766s CPU time.
░░ Subject: Resources consumed by unit runtime
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit elasticsearch.service completed and consumed the indicated resources.

I ran journalctl -u elasticsearch.service and got the below out put

systemd[1]: Starting Elasticsearch...
systemd[1]: Started Elasticsearch.
systemd[1]: Stopping Elasticsearch...
systemd[1]: elasticsearch.service: Deactivated successfully.
systemd[1]: Stopped Elasticsearch.
systemd[1]: elasticsearch.service: Consumed 1min 5.740s CPU time.
systemd[1]: Starting Elasticsearch...
systemd-entrypoint[4036]: Exception in thread "main" org.elasticsearch.common.settings.SettingsException: Failed to load settings from [elasticsearch.yml]
systemd-entrypoint[4036]:         at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1233)
systemd-entrypoint[4036]:         at org.elasticsearch.node.InternalSettingsPreparer.loadConfigWithSubstitutions(InternalSettingsPreparer.java:126)
systemd-entrypoint[4036]:         at org.elasticsearch.node.InternalSettingsPreparer.prepareEnvironment(InternalSettingsPreparer.java:52)
systemd-entrypoint[4036]:         at org.elasticsearch.common.cli.EnvironmentAwareCommand.createEnv(EnvironmentAwareCommand.java:110)
systemd-entrypoint[4036]:         at org.elasticsearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:54)
systemd-entrypoint[4036]:         at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:85)
systemd-entrypoint[4036]:         at org.elasticsearch.cli.Command.main(Command.java:50)
systemd-entrypoint[4036]:         at org.elasticsearch.launcher.CliToolLauncher.main(CliToolLauncher.java:64)
systemd-entrypoint[4036]: Caused by: org.elasticsearch.xcontent.XContentParseException: [109:29] Duplicate field 'cluster.initial_master_nodes'
systemd-entrypoint[4036]:  at [Source: (ByteArrayInputStream); line: 109, column: 29]
systemd-entrypoint[4036]:         at org.elasticsearch.xcontent.provider.json.JsonXContentParser.newXContentParseException(JsonXContentParser.java:52)
systemd-entrypoint[4036]:         at org.elasticsearch.xcontent.provider.json.JsonXContentParser.nextFieldName(JsonXContentParser.java:71)
systemd-entrypoint[4036]:         at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:759)
systemd-entrypoint[4036]:         at org.elasticsearch.common.settings.Settings.fromXContent(Settings.java:727)
systemd-entrypoint[4036]:         at org.elasticsearch.common.settings.Settings$Builder.loadFromStream(Settings.java:1229)
systemd-entrypoint[4036]:         ... 7 more
systemd-entrypoint[4036]: Caused by: com.fasterxml.jackson.core.JsonParseException: Duplicate field 'cluster.initial_master_nodes'
systemd-entrypoint[4036]:  at [Source: (ByteArrayInputStream); line: 109, column: 29]
systemd-entrypoint[4036]:         at com.fasterxml.jackson.core.json.JsonReadContext._checkDup(JsonReadContext.java:243)
systemd-entrypoint[4036]:         at com.fasterxml.jackson.core.json.JsonReadContext.setCurrentName(JsonReadContext.java:237)
systemd-entrypoint[4036]:         at com.fasterxml.jackson.dataformat.yaml.YAMLParser.nextToken(YAMLParser.java:482)
systemd-entrypoint[4036]:         at com.fasterxml.jackson.core.JsonParser.nextFieldName(JsonParser.java:1091)
systemd-entrypoint[4036]:         at org.elasticsearch.xcontent.provider.json.JsonXContentParser.nextFieldName(JsonXContentParser.java:69)
systemd-entrypoint[4036]:         ... 10 more

I noticed the following

Exception in thread "main" org.elasticsearch.common.settings.SettingsException: Failed to load settings from [elasticsearch.yml]

and

Caused by: org.elasticsearch.xcontent.XContentParseException: [109:29] Duplicate field 'cluster.initial_master_nodes'

There is only one 'cluster.initial_master_nodes' field in my elasticsearch.yml. I commented it out saved and restarted elasticsearch.service but Is still get the same error.

I have looked everywhere I can think of but I am out of ideas. I might even be on the wrong track...

I have had a look on the forums here and elsewhere but can't seem to find anything that anything that helped.

Can anyone help?

Thank you

DavidTurner · August 18, 2023, 6:16am

I think you're on the right track, Duplicate field 'cluster.initial_master_nodes' would definitely prevent Elasticsearch from starting. Note these docs indicating you shouldn't have it set at all:

IMPORTANT: After the cluster has formed, remove the cluster.initial_master_nodes setting from each node’s configuration. It should not be set for master-ineligible nodes, master-eligible nodes joining an existing cluster, or nodes which are restarting.

You can't get this error without genuinely having a duplicate cluster.initial_master_nodes setting. Perhaps you are not editing the config file that Elasticsearch is reading?

dirtdart666 · August 19, 2023, 3:01am

Hi David,

Thank you for the help and the link to the docs regarding cluster.initial_master_nodes.

I was editing /etc/elasticsearch/elasticsearch.yml That's the main config for Elasticsearch right?

I commented out cluster.initial_master_nodes configuration before but it didn't work.

I have obviously done something wrong somewhere...

I ended up rolling back the VM to a previous snapshot and starting again. This time I didn't add any cluster.initial_master_nodes`.' configuration and it works now.

Once again thanks for your help.

system · September 16, 2023, 3:02am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.