Fresh ELK setup.. no http?


#1

Following here: https://www.elastic.co/guide/en/elastic-stack/current/installing-elastic-stack.html And I've gotten elasticsearch, kibana, logstash and teh x-pack installed but no http when I hit teh servers IP. It's likely something trivial and stupid that I've missed but I can't find it.
Can ping the ip just fine, however nmap shows only 22 and 25 open. What'd I miss?


Kibana 5, Unregistered auth agent
(Magnus Bäck) #2

Did you forget do adjust the network.host option to have ES listen on non-loopback interfaces?


#3

Well I tried both the default 127.0.0.1 then when that didnt work I change it to the current IP on the only ethernet interface on this vm. Still go bueno.


(Magnus Bäck) #4

Have you verified that ES actually starts up?


#5

Hmm well it starts fine, until I try to hit the interface, then it bombs out with the same error I had yesterday:

[mtops@localhost elasticsearch]$ systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2016-11-01 09:41:46 MDT; 3s ago
Docs: http://www.elastic.co
Process: 2692 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet -Edefault.path.logs=${LOG_DIR} -Edefault.path.data=${DATA_DIR} -Edefault.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
Process: 2690 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 2692 (code=exited, status=1/FAILURE)

Nov 01 09:41:44 localhost.localdomain elasticsearch[2692]: 2016-11-01 09:41:43,988 main ERROR Null object returned for RollingFile in Appenders.
Nov 01 09:41:44 localhost.localdomain elasticsearch[2692]: 2016-11-01 09:41:43,989 main ERROR Null object returned for RollingFile in Appenders.
Nov 01 09:41:44 localhost.localdomain elasticsearch[2692]: 2016-11-01 09:41:43,989 main ERROR Unable to locate appender "rolling" for logger config "root"
Nov 01 09:41:44 localhost.localdomain elasticsearch[2692]: 2016-11-01 09:41:43,990 main ERROR Unable to locate appender "index_indexing_slowlog_rolling" for logger config "index.index...wlog.index"
Nov 01 09:41:44 localhost.localdomain elasticsearch[2692]: 2016-11-01 09:41:43,990 main ERROR Unable to locate appender "audit_rolling" for logger config "org.elasticsearch.xpack.secu...AuditTrail"
Nov 01 09:41:44 localhost.localdomain elasticsearch[2692]: 2016-11-01 09:41:43,995 main ERROR Unable to locate appender "index_search_slowlog_rolling" for logger config "index.search.slowlog"
Nov 01 09:41:44 localhost.localdomain elasticsearch[2692]: 2016-11-01 09:41:43,996 main ERROR Unable to locate appender "deprecation_rolling" for logger config "org.elasticsearch.deprecation"
Nov 01 09:41:46 localhost.localdomain systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 01 09:41:46 localhost.localdomain systemd[1]: Unit elasticsearch.service entered failed state.
Nov 01 09:41:46 localhost.localdomain systemd[1]: elasticsearch.service failed.
Hint: Some lines were ellipsized, use -l to show in full.

Yesterday it would not start at all until I commented out the x-pack line:

action.auto_create_index: .security,.monitoring*,.watches,.triggered_watches,.watcher-history*


#6

Can't find much on this but it seems to be Java related.


#7

Ok Magnus.. ignore the above entries. Those were indeed caused by a typo in the /var/log path. SO back to what I hope is the actual problem.

Now when starting ES and checking the status I get:

[mtops@localhost elasticsearch]$ systemctl status elasticsearch -l
● elasticsearch.service - Elasticsearch
Loaded: loaded (/usr/lib/systemd/system/elasticsearch.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2016-11-01 10:32:09 MDT; 14s ago
Docs: http://www.elastic.co
Process: 5916 ExecStart=/usr/share/elasticsearch/bin/elasticsearch -p ${PID_DIR}/elasticsearch.pid --quiet -Edefault.path.logs=${LOG_DIR} -Edefault.path.data=${DATA_DIR} -Edefault.path.conf=${CONF_DIR} (code=exited, status=1/FAILURE)
Process: 5914 ExecStartPre=/usr/share/elasticsearch/bin/elasticsearch-systemd-pre-exec (code=exited, status=0/SUCCESS)
Main PID: 5916 (code=exited, status=1/FAILURE)

Nov 01 10:31:52 localhost.localdomain systemd[1]: Starting Elasticsearch...
Nov 01 10:31:52 localhost.localdomain systemd[1]: Started Elasticsearch.
Nov 01 10:32:09 localhost.localdomain systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 01 10:32:09 localhost.localdomain systemd[1]: Unit elasticsearch.service entered failed state.
Nov 01 10:32:09 localhost.localdomain systemd[1]: elasticsearch.service failed.
[mtops@localhost elasticsearch]$

Stopping and restarting according to journalctl:

Nov 01 10:36:37 localhost.localdomain sudo[6578]: mtops : TTY=pts/1 ; PWD=/etc/elasticsearch ; USER=root ; COMMAND=/bin/systemctl start elasticsearch
Nov 01 10:36:37 localhost.localdomain polkitd[658]: Registered Authentication Agent for unix-process:6579:7437717 (system bus name :1.109 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8)
Nov 01 10:36:37 localhost.localdomain systemd[1]: Starting Elasticsearch...
Nov 01 10:36:37 localhost.localdomain systemd[1]: Started Elasticsearch.
Nov 01 10:36:37 localhost.localdomain polkitd[658]: Unregistered Authentication Agent for unix-process:6579:7437717 (system bus name :1.109, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_US.UTF-8) (disconnected from bus)
Nov 01 10:36:59 localhost.localdomain systemd[1]: elasticsearch.service: main process exited, code=exited, status=1/FAILURE
Nov 01 10:36:59 localhost.localdomain systemd[1]: Unit elasticsearch.service entered failed state.
Nov 01 10:36:59 localhost.localdomain systemd[1]: elasticsearch.service failed.


#8

Still having this weird issue. Updated Java to 8u111, no help.


(Mark Walkom) #9

Check your actual ES logs.


#10

/var/log/elasticsearch.log yes?


#11

Looking at this log, seems there's no entries since 10/31 when it was installed the last time.


#12

Did I find some unknown bug or just being an idiot and missing an obvious correction? Usually I'd go with the latter but people are typically all to quick to point out a mistake, lol.


#13

Seems this issue has grown stale. So I went looking for a resolution to my Kibana issue and found great help. It seems that Kibana 5 doesnt want to start unless it can read the ES url.

And since I followed the Elastic guide on setting up an ELK stack in version 5.0 my system has been broken. Here is a pastebin of my Es startup issue and a couple things I did to try to remedy it.

http://pastebin.com/0WEZ26eM


(Mark Walkom) #14

Depends, did you change cluster.name?

What do you mean?

Read the URL how? KB will start if if cannot connect to ES, it'll just sit there waiting for the cluster to become available if it's not. If you have a bad config setting, that may cause KB to not start.

Which guide?


#15

I did, it's changed to "gntc_elk"

That's me getting too anxious for a resolution.

Take a look at the pastebin link I embedded earlier.. That's kinda why I did that to show anyone what's going on.[quote="warkolm, post:14, topic:64576"]
And since I followed the Elastic guide on setting up an ELK stack in version 5.0 my system has been broken.
[/quote]

The url in my very first post in this thread. The official Elastic guide.


(Mark Walkom) #16

Ok, cause if so the log will be named accordingly, as it's based on that cluster name.

How did you install things though, using packages or tar?


#17

First time thru RPM's only. After I ran into issues I went for the tarballs.

I see about the log.. did not know that. Inspecting the contents though it's teh exact same messages as my last pastebin above. A bunch of java BS.


(system) #18