A quick search seems to indicate that if the command you are trying to execute is missing the executable flag, that can cause this error, and I know we've had broken packages released in the past. What version of Kibana are you trying to run, and where did you get the deb or rpm from?
Also note that port 9200 is for communicating with Elasticsearch, not Kibana. Kibana runs on port 5601.
Previous to 5.0, Elasticsearch did have a static webserver built in, which was used for something called site plugins, and is what a lot of web-based Elasticsearch plugins used. Kibana hasn't used it in 4.x or 5.x, and I don't even think it was used in 3.x either. Not sure about earlier.
Anyway, sounds like you have something running on 5601. Whether it's the Kibana instance or something else I couldn't tell you though. Does opening/curling http://localhost:5601 work? And if so, do you see Kibana or something else?
Whelp killing kibana stopped that port was listening. So Im going to try erasing and reinstalling it. Nothig comes up via a browser, and curl returns zilch.
Looks like Kibana won't start until I get my ES issues resolved..
[mtops@localhost bin]$ ./kibana
log [21:44:30.825] [info][status][plugin:kibana@5.0.0] Status changed from uninitialized to green - Ready
log [21:44:30.903] [info][status][plugin:elasticsearch@5.0.0] Status changed from uninitialized to yellow - Waiting for Elasticsearch
log [21:44:30.941] [error][elasticsearch] Request error, retrying
HEAD http://localhost:9200/ => connect ECONNREFUSED 127.0.0.1:9200
log [21:44:30.946] [info][status][plugin:console@5.0.0] Status changed from uninitialized to green - Ready
log [21:44:30.952] [warning][elasticsearch] Unable to revive connection: http://localhost:9200/
log [21:44:30.953] [warning][elasticsearch] No living connections
log [21:44:30.955] [error][status][plugin:elasticsearch@5.0.0] Status changed from yellow to red - Unable to connect to Elasticsearch at http://localhost:9200.
log [21:44:31.166] [info][status][plugin:timelion@5.0.0] Status changed from uninitialized to green - Ready
log [21:44:31.174] [info][listening] Server running at http://localhost:5601
log [21:44:31.175] [error][status][ui settings] Status changed from uninitialized to red - Elasticsearch plugin is red
log [21:44:33.464] [warning][elasticsearch] Unable to revive connection: http://localhost:9200/
log [21:44:33.465] [warning][elasticsearch] No living connections
log [21:44:35.970] [warning][elasticsearch] Unable to revive connection: http://localhost:9200/
log [21:44:35.971] [warning][elasticsearch] No living connections
log [21:44:38.483] [warning][elasticsearch] Unable to revive connection: http://localhost:9200/
log [21:44:38.485] [warning][elasticsearch] No living connections
Exception in thread "main" org.elasticsearch.bootstrap.BootstrapException: java.nio.file.NoSuchFileException: /usr/share/elasticsearch/config
That shouldn't be that way. Perhaps their package is broken? Are you using deb or rpm? And where'd you get the package from? I also don't see any mention of which OS you're on here.
So it sounds like you are trying to access the stack from another machine, which means you'll need to set the network.host setting in your elasticsearch.yml as was pointed out over there. Is there anything else you are changing in your config? I wonder if there's some typo or something that's presenting itself oddly here.
Looking through the trace you put in pastebin, it seems like maybe it's a permissions issue. It keep failing to read and write files, and you end up having to do things manually. What's different about your CentOS install then the way it comes stock?
Yeah sorry, I'm not a fan of multi posting but I need traction on this. I did set my network.host value. I tried both 'localhost', '127.0.0.1' and teh 'x.x.x.x' that we use internally. None of those changed the result.
As far as my CentOS basement layer, it's just a minimal install with epel-release and whatever yum decided needed updating and any pre-reqs for ES/KA or LS.
> # ======================== Elasticsearch Configuration =========================
> #
> # NOTE: Elasticsearch comes with reasonable defaults for most settings.
> # Before you set out to tweak and tune the configuration, make sure you
> # understand what are you trying to accomplish and the consequences.
> #
> # The primary way of configuring a node is via this file. This template lists
> # the most important settings you may want to configure for a production cluster.
> #
> # Please see the documentation for further information on configuration options:
> # <http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html>
> #
> # ---------------------------------- Cluster -----------------------------------
> #
> # Use a descriptive name for your cluster:
> #
> cluster.name: gntc_elk
> #
> # ------------------------------------ Node ------------------------------------
> #
> # Use a descriptive name for the node:
> #
> node.name: ${HOSTNAME}
> #
> # Add custom attributes to the node:
> #
> #node.attr.rack: r1
> #
> # ----------------------------------- Paths ------------------------------------
> #
> # Path to directory where to store the data (separate multiple locations by comma):
> #
> path.data: /var/log/data
> #
> # Path to log files:
> #
> path.logs: /var/log/elasticsearch
> #
> # ----------------------------------- Memory -----------------------------------
> #
> # Lock the memory on startup:
> #
> #bootstrap.memory_lock: true
> #
> # Make sure that the heap size is set to about half the memory available
> # on the system and that the owner of the process is allowed to use this
> # limit.
> #
> # Elasticsearch performs poorly when the system is swapping the memory.
> #
> # ---------------------------------- Network -----------------------------------
> #
> # Set the bind address to a specific IP (IPv4 or IPv6):
> #
> network.host: "127.0.0.1"
> #
> # Set a custom port for HTTP:
> #
> http.port: 9200
> #
> # For more information, see the documentation at:
> # <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html>
> #
> # --------------------------------- Discovery ----------------------------------
> #
> # Pass an initial list of hosts to perform discovery when new node is started:
> # The default list of hosts is ["127.0.0.1", "[::1]"]
> #
> #discovery.zen.ping.unicast.hosts: ["host1", "host2"]
> #
> # Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1):
> #
> #discovery.zen.minimum_master_nodes: 3
> #
> # For more information, see the documentation at:
> # <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html>
> #
> # ---------------------------------- Gateway -----------------------------------
> #
> # Block initial recovery after a full cluster restart until N nodes are started:
> #
> #gateway.recover_after_nodes: 3
> #
> # For more information, see the documentation at:
> # <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html>
> #
> # ---------------------------------- Various -----------------------------------
> #
> # Disable starting multiple nodes on a single system:
> #
> #node.max_local_storage_nodes: 1
> #
> # Require explicit names when deleting indices:
> #
> #action.destructive_requires_name: true
> #Below added for X-Pack
action.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher-history*
Ah, I see, you set the log paths yourself. I'm not super familiar with rpm or deb packages to be honest, but I believe that they create an elasticsearch user (and I think group too), so the file permissions makes sense here, since that user probably doesn't have access to those things by default.
Note that by setting the host to 127.0.0.1, you can only connect via that IP, so it will only be accessible internally on that machine. I know you said you also tried the external IP, which should work, I just wanted to point that out since it sounds like you are trying to connect over the network.
Aside from the file permissions, I don't see anything wrong, and there's really not much else that was changed here. My only suggestion would be to try running without those changes and see if you can at least get it to load. Everything should work together "out of the box", no configuration needed, and be accessible locally on that machine. Maybe it's easiest to start there, then add remote access via the network.host setting, and then start configuring log files.
So yes at some point, either installing the RPM or whatever an lasticsearch user and group was created. Now since I'm testing this, I ran a chmod 766 on /etc/elasticsearch/* and /usr/share/elasticsearch/* just to get rid of the pasky permissions issues.
Thanks for the clarification on the network.host, I changed that to teh internal IP that the eth1 interface is currently set to. Then I set a chmod 766 to /etc/elasticsearch/x-pack/* .
Now I think I read somewhere that calling the binary manually means you are required to define at least the paths.. if not other variables.
So a sudo systemctl start elasticsearch actually starts for about 10 seconds then stops itself. After making the network.host and perms change here's the log output
; USER=root ; COMMAND=/bin/systemctl stop elasticsearch
Nov 03 16:19:52 localhost.localdomain polkitd[658]: Registered Authentication Agent for unix-process:10252:26777175 (system bus name :1.256 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
Nov 03 16:19:52 localhost.localdomain systemd[1]: Stopped Elasticsearch.
Nov 03 16:19:52 localhost.localdomain polkitd[658]: Unregistered Authentication Agent for unix-process:10252:26777175 (system bus name :1.256, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
Nov 03 16:20:04 localhost.localdomain sudo[10259]: mtops : TTY=pts/0 ; PWD=/usr/share/elasticsearch ; USER=root ; COMMAND=/bin/systemctl start elasticsearch
Nov 03 16:20:04 localhost.localdomain polkitd[658]: Registered Authentication Agent for unix-process:10260:26778426 (system bus name :1.257 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
Nov 03 16:20:04 localhost.localdomain systemd[1]: Starting Elasticsearch...
Nov 03 16:20:04 localhost.localdomain systemd[1]: Started Elasticsearch.
Nov 03 16:24:56 localhost.localdomain polkitd[658]: Unregistered Authentication Agent for unix-process:10346:26807601 (system bus name :1.260, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
Nov 03 16:25:13 localhost.localdomain systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 03 16:25:13 localhost.localdomain systemd[1]: Unit elasticsearch.service entered failed state.
Nov 03 16:25:13 localhost.localdomain systemd[1]: elasticsearch.service failed.
Now I think I read somewhere that calling the binary manually means you are required to define at least the paths.. if not other variables.
That may be, I've never tried running the stack from RPM or DEB before. We have automated tests to make sure that setup works though, so if it's broken, I'd be pretty surprised.
Nov 03 16:20:04 localhost.localdomain polkitd[658]: Registered Authentication Agent for unix-process:10260:26778426 (system bus name :1.257 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
And there's that line again. So it's an issue with polkit (PolicyKit) complaining about... something.
I don't know anything about polkit but what I just learned searching around a little. It appears to be a way for unprivileged processes to access privileged services, but I'm not sure how it's being used here. Elasticsearch will run as a normal user just fine, but things change when you're running it with systemd, or any other system script.
I have no idea why it wouldn't load though. Maybe there's something missing from the package that is preventing it from working with pokit, or maybe there's something weird about your particular setup. Seems it's definitely being caused by polkit though, Elasticsearch doesn't even start up.
In any event, you should be able to run the command by hand. Looking back at your pastebin from before, it seems that the last time you tried that, it simply failed to write log files to /usr/share/elasticsearch/logs. Maybe it's worth trying to just chmod -r 777 /usr/share/elasticsearch/* just to see if you can get it to start up?
Interesting. So I did what you asked. I chmodded 777 for usr/share/elasticsearch/ *. I tried running the binary and got a couple errors about no data subfolder. Likely because I didn't call it using the right paths variables. Being lazy and not wanting to look up the correct syntax I created teh subfolder then re-called the bin and it actually fired up!
Now.. since teh supposed only differences between a manual and a daemon run are the paths called and what.. the yml? Something must be amiss with my config.
Now.. since teh supposed only differences between a manual and a daemon run are the paths called and what.. the yml? Something must be amiss with my config.
I don't know if it's your config, I think there's something going on with systemd or polkit, at least based on what I see in your output. I don't really have experience with either though, so I can't offer any advice there.
You did get some error output when you ran it by hand, specifically exception when exporting documents org.elasticsearch.xpack.monitoring.exporter.ExportException: exporters are either not ready or faulty. I don't know what that means though.
I have to assume elasticsearch teh user cannot access it. Doesn't make sense to me though. So I chown'd elasticsearch on /usr/share/elasticsearch/data
If you are running Elasticsearch as your user (ie. by running bin/elasticsearch yourself), your user is the one that needs access to /usr/share/elasticsearch/data, not the elasticsearch user. The elasticseach user would be used if you were using systemd to launch it. Still, the permissions are 777, so it should work.
Bouncing the service still fails.. but now I really have no clue why outside of teh PolicyKit1/AuthenticationAgent error.
Always back to square one. It really looks to me like something's up with polkit/PolicyKit here. I wish I could give you some more input, but I have no idea what the issue might be.
The only suggestion I have is to try asking in IRC using the elasticsearch channel on freenode, or maybe bringing this back to the thread in the Elasticsearch section and seeing if you get lucky and bump into someone that knows more about what's going on with the permissions and policy stuff.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
I got all packages from links in the official Elastic guide: 