Elasticsearch does not start because of a graylog-server error

uek · June 17, 2020, 3:06pm

Hello there,

my elasticsearch-oss I would like to use in combination with prometheus and grafana does not start on my centos8 machine because of theese errors :
Automatic restarting of the unit graylog-server.service has been scheduled, as the result for

-- the configured Restart= setting for the unit.*
Jun 17 16:55:21 keyhole systemd[1]: Stopped Graylog server.*
-- Subject: Unit graylog-server.service has finished shutting down*
-- Defined-By: systemd*
-- Support: https://access.redhat.com/support*
-- *
-- Unit graylog-server.service has finished shutting down.*
Jun 17 16:55:21 keyhole systemd[1]: Started Graylog server.*
-- Subject: Unit graylog-server.service has finished start-up*
-- Defined-By: systemd*
-- Support: https://access.redhat.com/support*
-- *
-- Unit graylog-server.service has finished starting up.*
-- *
-- The start-up result is done.*
Jun 17 16:55:25 keyhole systemd[1]: graylog-server.service: Main process exited, code=exited, status=1/FAILURE*
Jun 17 16:55:25 keyhole systemd[1]: graylog-server.service: Failed with result 'exit-code'.*

I already removed /var/lib/elasticsearch/nodes/0/node.lock
to abvoid an Exception

Caused by: java.io.IOException: failed to obtain lock on /var/lib/elasticsearch/nodes/0

, but without success elaticsearch does not start.
Please find my logfile here: https://pastebin.com/mkxiTDQm

the elasticsearch.yml here: https://pastebin.com/xWuDxJhk

If anyone has an idea or solution , please let me know.

Many thanks in advance

Uli

Vinayak_Sapre · June 18, 2020, 2:04am

This may be directory permission issue. For ex. if /var/lib/elasticsearch is owned by root and service is running as elasticsearch user.

Everything under /var/lib/elasticsearch should be owned / writable by the user running elasticsearch service.

uek · June 18, 2020, 9:55am

Hello Vinayak,

Thanks for your advice. I changed the permissions
chmod 777 -R elasticsearch:elasticsearch /var/lib/elasticsearch

but without success

uek1967 · June 18, 2020, 12:58pm

it now looks like: cd /var/lib/elasticsearch/

[root@keyhole elasticsearch]# ls -la*
total 4*
drwxrwsrwx. 3 elasticsearch elasticsearch 19 May 28 18:37 .*
drwxr-xr-x. 67 root root 4096 Jun 15 22:34 ..*
drwxrwsrwx. 3 elasticsearch elasticsearch 15 Jun 12 19:13 nodes*

but no change and nothing in elasticsearch.log

Any Advice is appreciated

Thanks in advance

Uli

Vinayak_Sapre · June 19, 2020, 7:17am

Hi Uli

I found Caused by: java.nio.file.NoSuchFileException: /var/lib/elasticsearch/nodes/0/node.lock in the logs.

Can you check if the file /var/lib/elasticsearch/nodes/0/node.lock exists?

If not try sudo touch /var/lib/elasticsearch/nodes/0/node.lock This will create a zero byte file.. Then change ownership of this file to elasticsearch and permissions to 644.

uek1967 · June 19, 2020, 12:54pm

Hi Vinayak,

I deleted that file following another advice und cerated ist now as you said:

touch /var/lib/elasticsearch/nodes/0/node.lock

and set permissions to 644
chmod 644 /var/lib/elasticsearch/nodes/0/node.lock

but it doesn't start

systemctl start elasticsearch.service
Job for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

Here I have the last 50 lines of the logfile ,

*tail -50 /var/log/elasticsearch/elasticsearch.log *
*		at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) [elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) [elasticsearch-cli-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) [elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.7.1.jar:7.7.1]*
*[2020-06-19T14:32:13,293][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [keyhole] uncaught exception in thread [main]*
*org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [nsport.host] please check that any required plugins are installed, or check the breaking changes documentation for removed settings*
*	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.7.1.jar:7.7.1]*
*Caused by: java.lang.IllegalArgumentException: unknown setting [nsport.host] please check that any required plugins are installed, or check the breaking changes documentation for removed settings*
*	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:544) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:460) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:431) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:149) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.node.Node.<init>(Node.java:365) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.node.Node.<init>(Node.java:264) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	... 6 more*
*	Suppressed: java.lang.IllegalArgumentException: unknown setting [xpack.ml.enabled] please check that any required plugins are installed, or check the breaking changes documentation for removed settings*
*		at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:544) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:460) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:431) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:149) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.node.Node.<init>(Node.java:365) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.node.Node.<init>(Node.java:264) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.7.1.jar:7.7.1]*

I found this:
uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [nsport.host] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

and this:
llegalArgumentException: unknown setting [nsport.host] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

futher this:

Suppressed: java.lang.IllegalArgumentException: unknown setting [xpack.ml.enabled] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

I changed elasticsearch.yml section like this:

*# Set the bind address to a specific IP (IPv4 or IPv6):*
*#*
*network.host: 127.0.0.1*
*#*
*# Set a custom port for HTTP:*
*#*
*http.port: 9200*

*#transsport.host: localhost  *

*#transport.tcp.port: 9300 *

*xpack.ml.enabled: false*

but all this didn't help to start elasticsearch

I'm pretty much at the end of my rope. Any thoughts on what I might try?

Many thanks in advance

Uli

Vinayak_Sapre · June 19, 2020, 2:27pm

Hi Uli
Looks like you have moved past the "lock" issue.

elasticsearch.yml you posted in your first post transport.host was misspelled as nsport.host. In the latest post, it is misspelled again transsport.host:.

Is * present as the first character on each line? If yes, that's an issue too.

Setting transport.host to localhost means you cannot expand the cluster as no other node will be able to reach this node on 9300. Comment out the line nsport.host. Also, if you may want to use special values for network.host as describe https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html

network.host: 192.168.10.3
#
# Set a custom port for HTTP:
#
http.port: 9200

#transport.host: localhost

transport.tcp.port: 9300

xpack.ml.enabled: false

uek1967 · June 19, 2020, 5:05pm

Hello Vinayak,

I am am terribly to inform you that the modifications in elasticsearch.ym didn't change anything. Same error as before.

I am trying to find a solution for this:

Caused by: java.lang.IllegalArgumentException: unknown setting [xpack.ml.enabled] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

If you maybe have another proposal please feel free to tell me.

Many thanks in advance

Uli

Vinayak_Sapre · June 19, 2020, 5:22pm

Uli,
If your setup has multiple issues it will take iterations to solve them one by one.

I just noticed in your first post you mentioned elasticsearch-oss. This version does not include x-pack plugin. So delete xpack.ml.enabled line from yaml.

uek1967 · June 19, 2020, 9:05pm

Hello Vinayak,

Thanks again; I removed xmlpack from the elasticsearch.ym , but elasticstearch still does not start.
Here is a plot of the logfile:
*tail -20 /var/log/elasticsearch/elasticsearch.log *
[2020-06-19T22:35:24,278][INFO ][o.e.p.PluginsService [2020-06-19T22:35:24,280][INFO ][o.e.p.PluginsService [2020-06-19T22:35:24,281][INFO ][o.e.p.PluginsService [2020-06-19T22:35:24,282][INFO ][o.e.p.PluginsService [2020-06-19T22:35:24,284][INFO ][o.e.p.PluginsService [2020-06-19T22:35:24,285][INFO ][o.e.p.PluginsService [2020-06-19T22:35:24,286][INFO ][o.e.p.PluginsService [2020-06-19T22:35:24,291][INFO ][o.e.p.PluginsService [2020-06-19T22:35:39,195][INFO ][o.e.d.DiscoveryModule [2020-06-19T22:35:41,276][INFO ][o.e.n.Node [2020-06-19T22:35:41,277][INFO ][o.e.n.Node [2020-06-19T22:35:41,858][INFO ][o.e.t.TransportService [2020-06-19T22:35:43,301][INFO ][o.e.b.BootstrapChecks [2020-06-19T22:35:43,315][ERROR][o.e.b.Bootstrap [1] bootstrap checks failed
[1]: the default discovery settings are unsuitable [2020-06-19T22:35:43,328][INFO ][o.e.n.Node [2020-06-19T22:35:43,390][INFO ][o.e.n.Node [2020-06-19T22:35:43,396][INFO ][o.e.n.Node [2020-06-19T22:35:43,482][INFO ][o.e.n.Node l
] [keyhole] loaded module [percolator]
] [keyhole] loaded module [rank-eval]
] [keyhole] loaded module [reindex]
] [keyhole] loaded module [repository-url]
] [keyhole] loaded module [systemd]
] [keyhole] loaded module [tasks]
] [keyhole] loaded module [transport-netty4]
] [keyhole] no plugins loaded
] [keyhole] using discovery type [zen] and seed hosts providers [settings]
] [keyhole] initialized
] [keyhole] starting ...
] [keyhole] publish_address {192.168.10.3:9300}, bound_addresses {192.168.10.3:9300}
] [keyhole] bound or publishing to a non-loopback address, enforcing bootstrap checks
] [keyhole] node validation exception
for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
] [keyhole] stopping ...
] [keyhole] stopped
] [keyhole] closing ...
] [keyhole] closed

Might it be this, that prevents elastisearch from starting:

*1] bootstrap checks failed*
*[1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured*

Thanks in advance,

Uli

Vinayak_Sapre · June 19, 2020, 10:23pm

I am assuming

This is your local development environment. Before setting up a shared or a production cluster you must read https://www.elastic.co/guide/en/elasticsearch/reference/7.8/setup.html.
You are not planning to add more nodes to the ES cluster. If you are, then you need to read setup guide.
You want this ES to be accessible from a host other than 192.168.10.3. If the application accessing ES is running on the same host (192.168.10.3), this is typical in a development environment, just comment the network.host line. Then other application can access it as http://localhost:9200

If these 3 assumptions are valid add following line to elasticsearch.yml
discovery.seed_hosts: 192.168.10.3

Then the other application will be able access it using http://192.168.10.3:9200

Must read
https://www.elastic.co/guide/en/elasticsearch/reference/7.8/network.host.html
https://www.elastic.co/guide/en/elasticsearch/reference/7.8/discovery-settings.html
https://www.elastic.co/guide/en/elasticsearch/reference/7.8/modules-network.html

uek1967 · June 23, 2020, 1:41pm

Hello Vinayak,

Many thanks again for your help. Following your advice I got elasticsearch and logstash running now. Neither filebeat nor metricbeat I was able to get running and instead of Kibana I will use prometheus and grafana. (bothe are running yet). What I didn't found out yet is how I have to edtit the .yml files so that my elasticsearch host monitors all devices in my network and waht is the logstash index that grafana asks for.

Best regards,

Uli

Vinayak_Sapre · June 29, 2020, 12:04am

Hi Uli,
If your original issues is solved, then you can mark this solved and open another topic.

uek1967 · June 29, 2020, 12:05pm

Problem solved!

Many thanks

system · July 27, 2020, 12:05pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.