New Installation Kibana 500 Internal Server Error

[KNXLabGeek (David Kirk)]

I am new to elastic search and having a problem narrowing down the issue. I have followed the install instructions verbatim and this is my 3rd attempt and after installing Kibana I hit the same issue. I am doing a new install of Elastic Search and Kibana (haven't gotten to installing Logstash). When I get to the http://knxlabelastic.knxlab.local/status/ I am getting 500 Internal Server Error.

if i test the ip address for nginx i get the Welcome to nginx! Screen.

I am installing it on Ubuntu 20.04. When I run systemctl status elasticsearch I get
'elasticsearch.service - Elasticsearch'
' Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)'
' Active: active (running) since Tue 2020-10-20 15:10:09 UTC; 1h 45min ago'
' Docs: https://www.elastic.co'
' Main PID: 810 (java)'
' Tasks: 59 (limit: 4621)'
' Memory: 1.3G'
' CGroup: /system.slice/elasticsearch.service'
' ├─ 810 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto' 'Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10' '-XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Dj>'
└─1215 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller'
'
Warning: some journal files were not opened due to insufficient permissions.'

'systemctl status kibana'
' kibana.service - Kibana'
' Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)'
' Active: active (running) since Tue 2020-10-20 15:09:23 UTC; 1h 47min ago'
' Main PID: 817 (node)'
' Tasks: 11 (limit: 4621)'
' Memory: 962.5M'
' CGroup: /system.slice/kibana.service'
' └─817 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli'

'Warning: some journal files were not opened due to insufficient permissions.'

'curl -XGET http://10.75.18.26:9200'
{
' "name" : "knxlabsyslog-01",'
' "cluster_name" : "elasticsearch",'
' "cluster_uuid" : "v-9wEFW9TpK37_GFu_RFOg",'
' "version" : {'
' "number" : "7.9.2",'
' "build_flavor" : "default",'
' "build_type" : "deb",'
' "build_hash" : "d34da0ea4a966c4e49417f2da2f244e3e97b4e6e",'
' "build_date" : "2020-09-23T00:45:33.626720Z",'
' "build_snapshot" : false,'
' "lucene_version" : "8.6.2",'
' "minimum_wire_compatibility_version" : "6.8.0",'
' "minimum_index_compatibility_version" : "6.0.0-beta1"'
' },'
' "tagline" : "You Know, for Search"'
'}'

It is a single node setup.
the only changes to elasticsearch.yml are:
'# Set the bind address to a specific IP (IPv4 or IPv6):'

'network.host: "10.75.18.26"'

Set a custom port for HTTP:

'http.port: 9200'

Changes to kibana.yml are:
'server.port: 5601'
'server.host: "10.75.18.26"'
'server.name: "knxlabelastic.knxlabs.local"'
'elasticsearch.hosts: ["http://10.75.18.26:9200"]'

'/etc/nginx/sites-available/KnxLabElastic.knxlabs.local'
'server {'
' listen 80;'

' server_name KnxLabElastic.knxlabs.local;'

' auth_basic "Restricted Access";'
' auth_basic_user_file /etc/nginx/htpasswd.users;'

' location / {'
' proxy_pass http://10.75.18.26:5601;'
' proxy_http_version 1.1;'
' proxy_set_header Upgrade $http_upgrade;'
' proxy_set_header Connection 'upgrade';'
' proxy_set_header Host $host;'
' proxy_cache_bypass $http_upgrade;'

any help and guidance would be appreciated.

[KNXLabGeek (David Kirk)]

I tried two other fixes with no change:
I added the line discovery.type: single-node to the elastisearch.yml and added policies to the firewall to allow access from a remote server.

[LeeDr (Lee Drengenberg)]

Hi David,

Have you found the issue with your install yet?

If not, can you look into the Elasticsearch log and let us know what you find there?

If you set your elasticsearch.yml network.host to anything other than localhost it will enforce bootstrap checks and might be logging a problem in the log (but I think that would usually make it shut down, not stay running).

Similarly for the kibana log.

Thanks,
Lee

[KNXLabGeek (David Kirk)]

Not yet. I have started from scratch 4 times with different instruction sets and run into the same issue each time.
from the log file:
[2020-11-03T01:30:00,001][INFO ][o.e.x.m.MlDailyMaintenanceService] [knxlabsyslog-01] triggering scheduled [ML] maintenance tasks
[2020-11-03T01:30:00,016][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [knxlabsyslog-01] Deleting expired data
[2020-11-03T01:30:00,018][INFO ][o.e.x.m.a.TransportDeleteExpiredDataAction] [knxlabsyslog-01] Completed deletion of expired ML data
[2020-11-03T01:30:00,018][INFO ][o.e.x.m.MlDailyMaintenanceService] [knxlabsyslog-01] Successfully completed [ML] maintenance tasks
[2020-11-03T01:30:00,011][INFO ][o.e.x.s.SnapshotRetentionTask] [knxlabsyslog-01] starting SLM retention snapshot cleanup task
[2020-11-03T01:30:00,018][INFO ][o.e.x.s.SnapshotRetentionTask] [knxlabsyslog-01] there are no repositories to fetch, SLM retention snapshot cleanup task complete

from the elasticsearch.yml

======================== Elasticsearch Configuration =========================

NOTE: Elasticsearch comes with reasonable defaults for most settings.

Before you set out to tweak and tune the configuration, make sure you

understand what are you trying to accomplish and the consequences.

The primary way of configuring a node is via this file. This template lists

the most important settings you may want to configure for a production cluster.

Please consult the documentation for further information on configuration options:

https://www.elastic.co/guide/en/elasticsearch/reference/index.html

---------------------------------- Cluster -----------------------------------

Use a descriptive name for your cluster:

#cluster.name: my-application

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

#node.name: node-1

Add custom attributes to the node:

#node.attr.rack: r1

----------------------------------- Paths ------------------------------------

Path to directory where to store the data (separate multiple locations by comma):

path.data: /var/lib/elasticsearch

Path to log files:

path.logs: /var/log/elasticsearch

----------------------------------- Memory -----------------------------------

Lock the memory on startup:

#bootstrap.memory_lock: true

Make sure that the heap size is set to about half the memory available

on the system and that the owner of the process is allowed to use this

limit.

Elasticsearch performs poorly when the system is swapping the memory.

---------------------------------- Network -----------------------------------

Set the bind address to a specific IP (IPv4 or IPv6):

network.host: "localhost"

Set a custom port for HTTP:

http.port: 9200

For more information, consult the network module documentation.

--------------------------------- Discovery ----------------------------------

Pass an initial list of hosts to perform discovery when this node is started:

The default list of hosts is ["127.0.0.1", "[::1]"]

#discovery.seed_hosts: ["host1", "host2"]

Bootstrap the cluster using an initial set of master-eligible nodes:

#cluster.initial_master_nodes: ["node-1", "node-2"]

For more information, consult the discovery and cluster formation module documentation.

discovery.type: single-node

---------------------------------- Gateway -----------------------------------

Block initial recovery after a full cluster restart until N nodes are started:

#gateway.recover_after_nodes: 3

For more information, consult the gateway module documentation.

---------------------------------- Various -----------------------------------

Require explicit names when deleting indices:

#action.destructive_requires_name: true

Kibana.log is empty

dkirk_admin@knxlabsyslog-01:~$ sudo systemctl status elasticsearch
● elasticsearch.service - Elasticsearch
Loaded: loaded (/lib/systemd/system/elasticsearch.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-10-22 15:36:50 UTC; 1 weeks 5 days ago
Docs: https://www.elastic.co
Main PID: 89347 (java)
Tasks: 61 (limit: 4621)
Memory: 1.3G
CGroup: /system.slice/elasticsearch.service
├─89347 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -D>
└─89536 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller

Oct 22 15:36:27 knxlabsyslog-01 systemd[1]: Starting Elasticsearch...
Oct 22 15:36:50 knxlabsyslog-01 systemd[1]: Started Elasticsearch.
dkirk_admin@knxlabsyslog-01:~$ sudo systemctl status kibana
● kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-10-22 15:37:11 UTC; 1 weeks 5 days ago
Main PID: 89593 (node)
Tasks: 11 (limit: 4621)
Memory: 653.2M
CGroup: /system.slice/kibana.service
└─89593 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli

Oct 22 15:37:36 knxlabsyslog-01 kibana[89593]: {"type":"log","@timestamp":"2020-10-22T15:37:36Z","tags":["status","plugin:elasticsearch@7.9.2","info"],"pid":89593,"state":"green","message":"Status changed from yellow to gre>
Oct 22 15:37:36 knxlabsyslog-01 kibana[89593]: {"type":"log","@timestamp":"2020-10-22T15:37:36Z","tags":["status","plugin:xpack_main@7.9.2","info"],"pid":89593,"state":"green","message":"Status changed from uninitialized to>
Oct 22 15:37:36 knxlabsyslog-01 kibana[89593]: {"type":"log","@timestamp":"2020-10-22T15:37:36Z","tags":["status","plugin:monitoring@7.9.2","info"],"pid":89593,"state":"green","message":"Status changed from uninitialized to>
Oct 22 15:37:36 knxlabsyslog-01 kibana[89593]: {"type":"log","@timestamp":"2020-10-22T15:37:36Z","tags":["status","plugin:spaces@7.9.2","info"],"pid":89593,"state":"green","message":"Status changed from uninitialized to gre>
Oct 22 15:37:36 knxlabsyslog-01 kibana[89593]: {"type":"log","@timestamp":"2020-10-22T15:37:36Z","tags":["status","plugin:security@7.9.2","info"],"pid":89593,"state":"green","message":"Status changed from uninitialized to g>
Oct 22 15:37:36 knxlabsyslog-01 kibana[89593]: {"type":"log","@timestamp":"2020-10-22T15:37:36Z","tags":["status","plugin:beats_management@7.9.2","info"],"pid":89593,"state":"green","message":"Status changed from uninitiali>
Oct 22 15:37:36 knxlabsyslog-01 kibana[89593]: {"type":"log","@timestamp":"2020-10-22T15:37:36Z","tags":["status","plugin:console_legacy@7.9.2","info"],"pid":89593,"state":"green","message":"Status changed from uninitialize>
Oct 22 15:37:36 knxlabsyslog-01 kibana[89593]: {"type":"log","@timestamp":"2020-10-22T15:37:36Z","tags":["status","plugin:apm_oss@7.9.2","info"],"pid":89593,"state":"green","message":"Status changed from uninitialized to gr>
Oct 22 15:37:36 knxlabsyslog-01 kibana[89593]: {"type":"log","@timestamp":"2020-10-22T15:37:36Z","tags":["listening","info"],"pid":89593,"message":"Server running at http://localhost:5601"}
Oct 22 15:37:37 knxlabsyslog-01 kibana[89593]: {"type":"log","@timestamp":"2020-10-22T15:37:37Z","tags":["info","http","server","Kibana"],"pid":89593,"message":"http server running at http://localhost:5601"}

[LeeDr (Lee Drengenberg)]

Hi David,
It would be helpful if you could use the code formatting features. It would make your yml files and logs files much easier to read.

Could you explain your purpose of using nginx ? Can you try without nginx to eliminate other problems?

I'm not an expert on nginx configuration but there's a lot of other posts I find with the subject;
https://discuss.elastic.co/search?q=nginx%20category%3A7%20order%3Alatest