First and foremost: Apologies, if this post is poorly formatted, i am unfamiliar with this editor. I will try to fix it once i see it, because i can not find the preview function.
Background: I am trying to get a Wazuh implementation running. Feel free to point me to their forum if this issue is a Wazuh-related problem, but it seems to me like this might be the better place. I am trying to get my Kibana Server to talk to my Elasticsearch Server. I had configured a setup with Kibana and Elasticsearch on one server before which worked fine. Now my Kibana server can not connect to my Elasticsearch server via port 9200 and my cluster of three Elasticsearch servers can not decide who the master server should be. I will start with the Kibana problem so i do not go over the character limit:
Here is the log output from my Kibana Server:
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["warning","config","deprecation"],"pid":1490,"message":"Setting [elasticsearch.username] to \"elastic\" is deprecated. You should use the \"kibana\" user instead."}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["warning","config","deprecation"],"pid":1490,"message":"Setting [xpack.monitoring.elasticsearch.username] to \"elastic\" is deprecated. You should use the \"kibana\" user instead."}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins-system"],"pid":1490,"message":"Setting up [37] plugins: [taskManager,siem,licensing,infra,encryptedSavedObjects,code,usageCollection,metrics,canvas,timelion,features,security,apm_oss,translations,r
eporting,uiActions,data,navigation,status_page,share,newsfeed,kibana_legacy,management,dev_tools,inspector,expressions,visualizations,embeddable,advancedUiActions,dashboard_embeddable_container,home,spaces,cloud,apm,graph,eui_utils,bfetch]"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","taskManager"],"pid":1490,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","siem"],"pid":1490,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","licensing"],"pid":1490,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","infra"],"pid":1490,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","encryptedSavedObjects"],"pid":1490,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["warning","plugins","encryptedSavedObjects","config"],"pid":1490,"message":"Generating a random key for xpack.encryptedSavedObjects.encryptionKey. To be able to decrypt encrypted saved objects attributes after restart, please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","code"],"pid":1490,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","usageCollection"],"pid":1490,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","metrics"],"pid":1490,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","canvas"],"pid":1490,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","timelion"],"pid":1490,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","features"],"pid":1490,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","security"],"pid":1490,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["warning","plugins","security","config"],"pid":1490,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["warning","plugins","security","config"],"pid":1490,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","apm_oss"],"pid":1490,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","translations"],"pid":1490,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","data"],"pid":1490,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","share"],"pid":1490,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","home"],"pid":1490,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","spaces"],"pid":1490,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","cloud"],"pid":1490,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","apm"],"pid":1490,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","graph"],"pid":1490,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","plugins","bfetch"],"pid":1490,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["info","savedobjects-service"],"pid":1490,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["error","elasticsearch","data"],"pid":1490,"message":"Request error, retrying\nGET http://192.168.1.91:9200/_xpack => socket hang up"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["error","elasticsearch","data"],"pid":1490,"message":"Request error, retrying\nHEAD http://192.168.1.91:9200/.apm-agent-configuration => socket hang up"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["error","elasticsearch","admin"],"pid":1490,"message":"Request error, retrying\nGET http://192.168.1.91:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => socket hang up"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["warning","elasticsearch","data"],"pid":1490,"message":"Unable to revive connection: http://192.168.1.91:9200/"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["warning","elasticsearch","data"],"pid":1490,"message":"No living connections"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["warning","plugins","licensing"],"pid":1490,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["warning","elasticsearch","data"],"pid":1490,"message":"Unable to revive connection: http://192.168.1.91:9200/"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["warning","elasticsearch","data"],"pid":1490,"message":"No living connections"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["warning","elasticsearch","admin"],"pid":1490,"message":"Unable to revive connection: http://192.168.1.91:9200/"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["warning","elasticsearch","admin"],"pid":1490,"message":"No living connections"}
{"type":"log","@timestamp":"2020-06-16T14:21:49Z","tags":["error","savedobjects-service"],"pid":1490,"message":"Unable to retrieve version information from Elasticsearch nodes."}
{"type":"log","@timestamp":"2020-06-16T14:21:52Z","tags":["warning","elasticsearch","admin"],"pid":1490,"message":"Unable to revive connection: http://192.168.1.91:9200/"}
{"type":"log","@timestamp":"2020-06-16T14:21:52Z","tags":["warning","elasticsearch","admin"],"pid":1490,"message":"No living connections"}
{"type":"log","@timestamp":"2020-06-16T14:21:54Z","tags":["warning","elasticsearch","admin"],"pid":1490,"message":"Unable to revive connection: http://192.168.1.91:9200/"}
{"type":"log","@timestamp":"2020-06-16T14:21:54Z","tags":["warning","elasticsearch","admin"],"pid":1490,"message":"No living connections"}
{"type":"log","@timestamp":"2020-06-16T14:21:57Z","tags":["warning","elasticsearch","admin"],"pid":1490,"message":"Unable to revive connection: http://192.168.1.91:9200/"}
{"type":"log","@timestamp":"2020-06-16T14:21:57Z","tags":["warning","elasticsearch","admin"],"pid":1490,"message":"No living connections"}
{"type":"log","@timestamp":"2020-06-16T14:21:59Z","tags":["warning","elasticsearch","admin"],"pid":1490,"message":"Unable to revive connection: http://192.168.1.91:9200/"}
{"type":"log","@timestamp":"2020-06-16T14:21:59Z","tags":["warning","elasticsearch","admin"],"pid":1490,"message":"No living connections"}
{"type":"log","@timestamp":"2020-06-16T14:22:02Z","tags":["warning","elasticsearch","admin"],"pid":1490,"message":"Unable to revive connection: http://192.168.1.91:9200/"}
Here is the output of netstat -tulpen on the elasticsearch master node
Proto Recv-Q Send-Q Local Address Foreign Address State Benutzer Inode PID/Program name
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 0 13822 407/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 0 14202 563/master
tcp 0 0 0.0.0.0:5666 0.0.0.0:* LISTEN 109 13734 336/nrpe
tcp6 0 0 192.168.1.91:9200 :::* LISTEN 111 37614 2610/java
tcp6 0 0 192.168.1.91:9300 :::* LISTEN 111 37480 2610/java
tcp6 0 0 ::1:25 :::* LISTEN 0 14203 563/master
udp 0 0 127.0.0.1:323 0.0.0.0:* 0 13539 348/chronyd
udp 0 0 0.0.0.0:41297 0.0.0.0:* 0 13009 310/rsyslogd
Here is what i get, when i try to open a telnet connection from the kibana server to the elasticsearch master node with telnet 192.168.1.91 9200. It looks to me like the port is open.
Trying 192.168.1.91...
Connected to 192.168.1.91.
Escape character is '^]'.
Any help would be greatly appreciated! If you need further input from me, let me know.
Thanks!