Starting elasticsearch, PolicyKit authentication error

Elastic Stack Elasticsearch
1

I've had a few posts here already, but to cover the bases I'm trying to setup teh ELK stack on a fresh CentOS 7 minimal VM. I've been following the official guide: https://www.elastic.co/guide/en/elastic-stack/current/installing-elastic-stack.html and after resolving various other smaller issues I've returned to this one that's persisted through all others. I'm getting this error when I try to start ES via systemd.

Nov 04 14:04:02 localhost.localdomain sudo[2361]:    mtops : TTY=pts/0 ; PWD=/usr/share/elasticsearch/logs ; USER=root ; COMMAND=/bin/systemctl start elasticsearch
Nov 04 14:04:02 localhost.localdomain polkitd[677]: Registered Authentication Agent for unix-process:2362:134606 (system bus name :1.28 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/Authenti
Nov 04 14:04:02 localhost.localdomain systemd[1]: Starting Elasticsearch...
-- Subject: Unit elasticsearch.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has begun starting up.
Nov 04 14:04:02 localhost.localdomain systemd[1]: Started Elasticsearch.
-- Subject: Unit elasticsearch.service has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit elasticsearch.service has finished starting up.
--
-- The start-up result is done.
Nov 04 14:04:02 localhost.localdomain polkitd[677]: Unregistered Authentication Agent for unix-process:2362:134606 (system bus name :1.28, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnecte
Nov 04 14:04:13 localhost.localdomain systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 04 14:04:13 localhost.localdomain systemd[1]: Unit elasticsearch.service entered failed state.
Nov 04 14:04:13 localhost.localdomain systemd[1]: elasticsearch.service failed.
2

Hey,

I had never heard of policykit, so I looked it up and now I am even more confused. It seems to be a tool to grant certain priviliges to processes, which are usually only available for root.

Elasticsearch is explicitely configured to not run as root, so maybe Elasticsearch exits, after this mechanism kicked in. Can you check /var/log/elasticsearch/ to verify?

Did you change any of those policykit configurations by yourself?

Also, is this a VM created from one of the publicly available vagrant images, so one could reproduce this?

--Alex

3

No sir.. I wouldnt even know how! This VM is a normal CentoS 7 minimal build in ESXi. The only things installed aside from the requirements for ELK is nano and in the past two weeks.. two yum updates.

Right now, as of today this is my ES's current status started manually:

[mtops@localhost bin]$ ./elasticsearch
[2016-11-07T10:00:53,278][INFO ][o.e.n.Node               ] [] initializing ...
[2016-11-07T10:00:53,782][INFO ][o.e.e.NodeEnvironment    ] [gRVH-wV] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [44.5gb], net total_space [47.4gb], spins? [unknown], types [rootfs]
[2016-11-07T10:00:53,783][INFO ][o.e.e.NodeEnvironment    ] [gRVH-wV] heap size [1.9gb], compressed ordinary object pointers [true]
[2016-11-07T10:00:53,786][INFO ][o.e.n.Node               ] [gRVH-wV] node name [gRVH-wV] derived from node ID; set [node.name] to override
[2016-11-07T10:00:53,833][INFO ][o.e.n.Node               ] [gRVH-wV] version[5.0.0], pid[5546], build[253032b/2016-10-26T04:37:51.531Z], OS[Linux/3.10.0-327.36.2.el7.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_111/25.111-b14]
[2016-11-07T10:00:58,477][INFO ][o.e.p.PluginsService     ] [gRVH-wV] loaded module [aggs-matrix-stats]
[2016-11-07T10:00:58,477][INFO ][o.e.p.PluginsService     ] [gRVH-wV] loaded module [ingest-common]
[2016-11-07T10:00:58,478][INFO ][o.e.p.PluginsService     ] [gRVH-wV] loaded module [lang-expression]
[2016-11-07T10:00:58,478][INFO ][o.e.p.PluginsService     ] [gRVH-wV] loaded module [lang-groovy]
[2016-11-07T10:00:58,478][INFO ][o.e.p.PluginsService     ] [gRVH-wV] loaded module [lang-mustache]
[2016-11-07T10:00:58,478][INFO ][o.e.p.PluginsService     ] [gRVH-wV] loaded module [lang-painless]
[2016-11-07T10:00:58,478][INFO ][o.e.p.PluginsService     ] [gRVH-wV] loaded module [percolator]
[2016-11-07T10:00:58,479][INFO ][o.e.p.PluginsService     ] [gRVH-wV] loaded module [reindex]
[2016-11-07T10:00:58,479][INFO ][o.e.p.PluginsService     ] [gRVH-wV] loaded module [transport-netty3]
[2016-11-07T10:00:58,479][INFO ][o.e.p.PluginsService     ] [gRVH-wV] loaded module [transport-netty4]
[2016-11-07T10:00:58,480][INFO ][o.e.p.PluginsService     ] [gRVH-wV] no plugins loaded
[2016-11-07T10:01:05,720][INFO ][o.e.n.Node               ] [gRVH-wV] initialized
[2016-11-07T10:01:05,720][INFO ][o.e.n.Node               ] [gRVH-wV] starting ...
[2016-11-07T10:01:06,094][INFO ][o.e.t.TransportService   ] [gRVH-wV] publish_address {10.0.20.178:9300}, bound_addresses {10.0.20.178:9300}
[2016-11-07T10:01:06,102][INFO ][o.e.b.BootstrapCheck     ] [gRVH-wV] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
ERROR: bootstrap checks failed
max file descriptors [4096] for elasticsearch process likely too low, increase to at least [65536]
[2016-11-07T10:01:06,113][INFO ][o.e.n.Node               ] [gRVH-wV] stopping ...
[2016-11-07T10:01:06,164][INFO ][o.e.n.Node               ] [gRVH-wV] stopped
[2016-11-07T10:01:06,165][INFO ][o.e.n.Node               ] [gRVH-wV] closing ...
[2016-11-07T10:01:06,183][INFO ][o.e.n.Node               ] [gRVH-wV] closed
[mtops@localhost bin]$

And via systemctl:

[mtops@localhost bin]$ systemctl start elasticsearch
==== AUTHENTICATING FOR org.freedesktop.systemd1.manage-units ===
Authentication is required to manage system services or units.
Authenticating as: mtops
Password:
==== AUTHENTICATION COMPLETE ===
Failed to start elasticsearch.service: Access denied
[mtops@localhost bin]$ sudo systemctl start elasticsearch
Failed to start elasticsearch.service: Access denied
[mtops@localhost bin]$
4

~/elasticsearch-5.0.0/logs/elasticsearch.log:

http://pastebin.com/EB3NtQXZ

5

Getting kinda fed up with this install.. I think Im going to delete this vm and start over. Spinscale: is there any better or alternative guide to use than here: https://www.elastic.co/guide/en/elastic-stack/current/installing-elastic-stack.html ?

7

Nope Houss I never did get this resolved. Did not get any further help from the people who'd know so I abandoned the project and went with GrayLog. I tried at least 4 times, each time blowing away my VM's and starting over. This BS error occurred each time I tried setting up ElasticSearch manually.

8

Well I don't know if we have the exact same issue but I resolved it by giving full rights to the folders and all files in it (/etc/elasticsearch) to the user starting elasticsearch.

I understand your frustration, but I strongly advise you consider moving to ELK in the end, or at least use graylog alongside Kibana, because Graylog vizualisation is really, really lacking, if not pointless.

Edit : and graylog isn't compatible with the 5.x versions of ELK, so you will have to use older ones.

9

Thanks Houss.. I might re-trl ELK in the future but Im the new guy at work and Ive spent too much time already spinning my wheels.
I know I chmod'd a lot in trying to figure this out, IDK if I tried what you did or not though.