Elasticsearch does not start because of a graylog-server error

Elastic Stack Elasticsearch
1

Hello there,

my elasticsearch-oss I would like to use in combination with prometheus and grafana does not start on my centos8 machine because of theese errors :
Automatic restarting of the unit graylog-server.service has been scheduled, as the result for

  • -- the configured Restart= setting for the unit.*
  • Jun 17 16:55:21 keyhole systemd[1]: Stopped Graylog server.*
  • -- Subject: Unit graylog-server.service has finished shutting down*
  • -- Defined-By: systemd*
  • -- Support: https://access.redhat.com/support*
  • -- *
  • -- Unit graylog-server.service has finished shutting down.*
  • Jun 17 16:55:21 keyhole systemd[1]: Started Graylog server.*
  • -- Subject: Unit graylog-server.service has finished start-up*
  • -- Defined-By: systemd*
  • -- Support: https://access.redhat.com/support*
  • -- *
  • -- Unit graylog-server.service has finished starting up.*
  • -- *
  • -- The start-up result is done.*
  • Jun 17 16:55:25 keyhole systemd[1]: graylog-server.service: Main process exited, code=exited, status=1/FAILURE*
  • Jun 17 16:55:25 keyhole systemd[1]: graylog-server.service: Failed with result 'exit-code'.*

I already removed /var/lib/elasticsearch/nodes/0/node.lock
to abvoid an Exception

Caused by: java.io.IOException: failed to obtain lock on /var/lib/elasticsearch/nodes/0

, but without success elaticsearch does not start.
Please find my logfile here: https://pastebin.com/mkxiTDQm

the elasticsearch.yml here: https://pastebin.com/xWuDxJhk

If anyone has an idea or solution , please let me know.

Many thanks in advance

Uli

2

This may be directory permission issue. For ex. if /var/lib/elasticsearch is owned by root and service is running as elasticsearch user.

Everything under /var/lib/elasticsearch should be owned / writable by the user running elasticsearch service.

3

Hello Vinayak,

Thanks for your advice. I changed the permissions
chmod 777 -R elasticsearch:elasticsearch /var/lib/elasticsearch

but without success

4

it now looks like: cd /var/lib/elasticsearch/

  • [root@keyhole elasticsearch]# ls -la*
  • total 4*
  • drwxrwsrwx. 3 elasticsearch elasticsearch 19 May 28 18:37 .*
  • drwxr-xr-x. 67 root root 4096 Jun 15 22:34 ..*
  • drwxrwsrwx. 3 elasticsearch elasticsearch 15 Jun 12 19:13 nodes*

but no change and nothing in elasticsearch.log

Any Advice is appreciated

Thanks in advance

Uli

5

Hi Uli

I found Caused by: java.nio.file.NoSuchFileException: /var/lib/elasticsearch/nodes/0/node.lock in the logs.

Can you check if the file /var/lib/elasticsearch/nodes/0/node.lock exists?

If not try sudo touch /var/lib/elasticsearch/nodes/0/node.lock This will create a zero byte file.. Then change ownership of this file to elasticsearch and permissions to 644.

6

Hi Vinayak,

I deleted that file following another advice und cerated ist now as you said:

touch /var/lib/elasticsearch/nodes/0/node.lock

and set permissions to 644
chmod 644 /var/lib/elasticsearch/nodes/0/node.lock

but it doesn't start

systemctl start elasticsearch.service
Job for elasticsearch.service failed because the control process exited with error code.
See "systemctl status elasticsearch.service" and "journalctl -xe" for details.

Here I have the last 50 lines of the logfile ,

*tail -50 /var/log/elasticsearch/elasticsearch.log *
*		at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) [elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) [elasticsearch-cli-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) [elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.7.1.jar:7.7.1]*
*[2020-06-19T14:32:13,293][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [keyhole] uncaught exception in thread [main]*
*org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [nsport.host] please check that any required plugins are installed, or check the breaking changes documentation for removed settings*
*	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.7.1.jar:7.7.1]*
*Caused by: java.lang.IllegalArgumentException: unknown setting [nsport.host] please check that any required plugins are installed, or check the breaking changes documentation for removed settings*
*	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:544) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:460) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:431) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:149) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.node.Node.<init>(Node.java:365) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.node.Node.<init>(Node.java:264) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.7.1.jar:7.7.1]*
*	... 6 more*
*	Suppressed: java.lang.IllegalArgumentException: unknown setting [xpack.ml.enabled] please check that any required plugins are installed, or check the breaking changes documentation for removed settings*
*		at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:544) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:489) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:460) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.common.settings.AbstractScopedSettings.validate(AbstractScopedSettings.java:431) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.common.settings.SettingsModule.<init>(SettingsModule.java:149) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.node.Node.<init>(Node.java:365) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.node.Node.<init>(Node.java:264) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:127) ~[elasticsearch-cli-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.7.1.jar:7.7.1]*
*		at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.7.1.jar:7.7.1]*

I found this:
uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException: unknown setting [nsport.host] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

and this:
llegalArgumentException: unknown setting [nsport.host] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

futher this:

Suppressed: java.lang.IllegalArgumentException: unknown setting [xpack.ml.enabled] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

I changed elasticsearch.yml section like this:

*# Set the bind address to a specific IP (IPv4 or IPv6):*
*#*
*network.host: 127.0.0.1*
*#*
*# Set a custom port for HTTP:*
*#*
*http.port: 9200*

*#transsport.host: localhost  *

*#transport.tcp.port: 9300 *

*xpack.ml.enabled: false*

but all this didn't help to start elasticsearch

I'm pretty much at the end of my rope. Any thoughts on what I might try?

Many thanks in advance

Uli

7

Hi Uli
Looks like you have moved past the "lock" issue.

elasticsearch.yml you posted in your first post transport.host was misspelled as nsport.host. In the latest post, it is misspelled again transsport.host:.

Is * present as the first character on each line? If yes, that's an issue too.

Setting transport.host to localhost means you cannot expand the cluster as no other node will be able to reach this node on 9300. Comment out the line nsport.host. Also, if you may want to use special values for network.host as describe https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html

network.host: 192.168.10.3
#
# Set a custom port for HTTP:
#
http.port: 9200

#transport.host: localhost

transport.tcp.port: 9300

xpack.ml.enabled: false
8

Hello Vinayak,

I am am terribly to inform you that the modifications in elasticsearch.ym didn't change anything. Same error as before.

I am trying to find a solution for this:

Caused by: java.lang.IllegalArgumentException: unknown setting [xpack.ml.enabled] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

If you maybe have another proposal please feel free to tell me.

Many thanks in advance

Uli

9

Uli,
If your setup has multiple issues it will take iterations to solve them one by one.

I just noticed in your first post you mentioned elasticsearch-oss. This version does not include x-pack plugin. So delete xpack.ml.enabled line from yaml.

10

Hello Vinayak,

Thanks again; I removed xmlpack from the elasticsearch.yml
, but elasticstearch still does not start.
Here is a plot of the logfile:
*tail -20 /var/log/elasticsearch/elasticsearch.log *
[2020-06-19T22:35:24,278][INFO ][o.e.p.PluginsService ] [keyhole] loaded module [percolator]
[2020-06-19T22:35:24,280][INFO ][o.e.p.PluginsService ] [keyhole] loaded module [rank-eval]
[2020-06-19T22:35:24,281][INFO ][o.e.p.PluginsService ] [keyhole] loaded module [reindex]
[2020-06-19T22:35:24,282][INFO ][o.e.p.PluginsService ] [keyhole] loaded module [repository-url]
[2020-06-19T22:35:24,284][INFO ][o.e.p.PluginsService ] [keyhole] loaded module [systemd]
[2020-06-19T22:35:24,285][INFO ][o.e.p.PluginsService ] [keyhole] loaded module [tasks]
[2020-06-19T22:35:24,286][INFO ][o.e.p.PluginsService ] [keyhole] loaded module [transport-netty4]
[2020-06-19T22:35:24,291][INFO ][o.e.p.PluginsService ] [keyhole] no plugins loaded
[2020-06-19T22:35:39,195][INFO ][o.e.d.DiscoveryModule ] [keyhole] using discovery type [zen] and seed hosts providers [settings]
[2020-06-19T22:35:41,276][INFO ][o.e.n.Node ] [keyhole] initialized
[2020-06-19T22:35:41,277][INFO ][o.e.n.Node ] [keyhole] starting ...
[2020-06-19T22:35:41,858][INFO ][o.e.t.TransportService ] [keyhole] publish_address {192.168.10.3:9300}, bound_addresses {192.168.10.3:9300}
[2020-06-19T22:35:43,301][INFO ][o.e.b.BootstrapChecks ] [keyhole] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-06-19T22:35:43,315][ERROR][o.e.b.Bootstrap ] [keyhole] node validation exception
[1] bootstrap checks failed
[1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
[2020-06-19T22:35:43,328][INFO ][o.e.n.Node ] [keyhole] stopping ...
[2020-06-19T22:35:43,390][INFO ][o.e.n.Node ] [keyhole] stopped
[2020-06-19T22:35:43,396][INFO ][o.e.n.Node ] [keyhole] closing ...
[2020-06-19T22:35:43,482][INFO ][o.e.n.Node ] [keyhole] closed

Might it be this, that prevents elastisearch from starting:

*1] bootstrap checks failed*
*[1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured*

Thanks in advance,

Uli

11

I am assuming

  • This is your local development environment. Before setting up a shared or a production cluster you must read https://www.elastic.co/guide/en/elasticsearch/reference/7.8/setup.html.

  • You are not planning to add more nodes to the ES cluster. If you are, then you need to read setup guide.

  • You want this ES to be accessible from a host other than 192.168.10.3. If the application accessing ES is running on the same host (192.168.10.3), this is typical in a development environment, just comment the network.host line. Then other application can access it as http://localhost:9200

If these 3 assumptions are valid add following line to elasticsearch.yml
discovery.seed_hosts: 192.168.10.3

Then the other application will be able access it using http://192.168.10.3:9200

Must read
https://www.elastic.co/guide/en/elasticsearch/reference/7.8/network.host.html
https://www.elastic.co/guide/en/elasticsearch/reference/7.8/discovery-settings.html
https://www.elastic.co/guide/en/elasticsearch/reference/7.8/modules-network.html

12

Hello Vinayak,

Many thanks again for your help. Following your advice I got elasticsearch and logstash running now. Neither filebeat nor metricbeat I was able to get running and instead of Kibana I will use prometheus and grafana. (bothe are running yet). What I didn't found out yet is how I have to edtit the .yml files so that my elasticsearch host monitors all devices in my network and waht is the logstash index that grafana asks for.

Best regards,

Uli

13

Hi Uli,
If your original issues is solved, then you can mark this solved and open another topic.

14

Problem solved!

Many thanks

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.