Having issue with kibana

Hi, I'm new to this and have setup graylog and elasticsearch but having issue when setting up kibana.

Getting this error when I attempt to connect to kibana:

Mar 19 15:51:53 greylog kibana[7157]: {"type":"log","@timestamp":"2020-03-19T15:51:53Z","tags":["license","debug","xpack"],"pid":7157,"message":"Calling [data] Elasticsearch _xpack API. Polling frequency: 30001"}
Mar 19 15:51:53 greylog kibana[7157]: {"type":"log","@timestamp":"2020-03-19T15:51:53Z","tags":["license","warning","xpack"],"pid":7157,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [invalid_index_name_exception] Invalid index name [xpack], must not start with ''., with { index_uuid="na" & index="_xpack" } :: {"path":"/_xpack","statusCode":400,"response":"{\"error\":{\"root_cause\":[{\"type\":\"invalid_index_name_exception\",\"reason\":\"Invalid index name [xpack], must not start with ''.\",\"index_uuid\":\"na\",\"index\":\"_xpack\"}],\"type\":\"invalid_index_name_exception\",\"reason\":\"Invalid index name [xpack], must not start with ''.\",\"index_uuid\":\"na\",\"index\":\"_xpack\"},\"status\":400}"}"}

On the GUI, I get this error:
Cannot connect to the Elasticsearch cluster currently configured for Kibana.

elasticsearch.yml file output (please note: I took out the comment when pasting it)

cluster.name: graylog
action.auto_create_index: false
#node.name: node-1
#node.attr.rack: r1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
#bootstrap.memory_lock: true
network.host: localhost
http.port: 9200
#discovery.zen.minimum_master_nodes:
#gateway.recover_after_nodes: 3
#action.destructive_requires_name: true

kibana.yml output file.

server.port: 5601
server.host: 10.40.21.3
#server.basePath: ""
#server.rewriteBasePath: false
#server.maxPayloadBytes: 1048576
#server.name: "your-hostname"
elasticsearch.hosts: ["http://localhost:9200"]
#elasticsearch.preserveHost: true
#kibana.index: ".kibana"
#kibana.defaultAppId: "home"
#elasticsearch.username: "user"
#elasticsearch.password: "pass"
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]
#elasticsearch.ssl.verificationMode: full
#elasticsearch.pingTimeout: 1500
#elasticsearch.requestTimeout: 30000
#elasticsearch.requestHeadersWhitelist: [ authorization ]
#elasticsearch.customHeaders: {}
#elasticsearch.shardTimeout: 30000
#elasticsearch.startupTimeout: 5000
#elasticsearch.logQueries: false
#pid.file: /var/run/kibana.pid
logging.dest: stdout
#logging.silent: false
#logging.quiet: false
logging.verbose: true
#ops.interval: 5000
#i18n.locale: "en"

curl -XGET 'http://localhost:9200'
{
"name" : "dMAWY0g",
"cluster_name" : "graylog",
"cluster_uuid" : "Nd6pAnAxRVq6uJspvSVi2w",
"version" : {
"number" : "6.8.7",
"build_flavor" : "oss",
"build_type" : "rpm",
"build_hash" : "c63e621",
"build_date" : "2020-02-26T14:38:01.193138Z",
"build_snapshot" : false,
"lucene_version" : "7.7.2",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}

when i execute this, curl -XGET 'http://10.40.21.3:5601', nothing is returned.

I'm doing this on centos 7.

Btw, another topic but dont see logs under /var/log for kibana. I did a search as well. Not sure why that is the case so had to use journalctl -u kibana.service

Versions installed are:
elasticsearch: "6.8.7"
kibana: kibana-6.8.7-1.x86_64

Your help is appreciated. Thank you very much.

Are you using the open source or the default distribution of Elasticsearch with Kibana? You might see this error if you try to use the open source distribution of Elasticsearch with the default distribution of Kibana. The distribution flavors of Kibana and Elasticsearch must match.

I followed https://docs.graylog.org/en/3.2/pages/installation/os/centos.html to install elasticsearch. Once that was successful, I created repo for kibana and installed it as below.

cat kibana.repo
[kibana-6.x]
name=Kibana repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

yum install kibana.

So I guess I'm using an open source? What do I need to do to get it working? Basically, I'll be using elasticsearch for graylog application. I'd thought as long as the version were the same, I would be good? What do you suggest and also if you can please provide a link to reinstall from the distribution of ES with Kibana.

I'm not familliar with graylog, but I think you'll need to change your baseurl for the Kibana to baseurl=https://artifacts.elastic.co/packages/oss-6.x/yum. Otherwise, I think it's installing the default distribution of Kibana.

Thanks. you. I changed the baseurl as seen below.
[kibana-6.x]
name=Kibana repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/oss-6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md

========================
Got an error when attempting to yum install. It looks like it can't find the rpm file.

https://artifacts.elastic.co/packages/oss-6.x/yum/6.8.7/kibana-6.8.7-x86_64.rpm: [Errno 14] HTTPS Error 404 - Not Found

I went to the website and didn't see that file either. Do you know another baseurl I can use?

Thanks

Hi Mike,

Sorry, looks like there is some issue with that baseUrl. I'll check on it with the appropriate team here at Elastic.

Meanwhile, here is the downloadable rpm file for the open source distribution of Kibana. Perhaps you can use that in the meantime.
https://artifacts.elastic.co/downloads/kibana/kibana-oss-6.8.7-x86_64.rpm

Also, if you do not need to run an open source version of the software, you can use our default distribution for free forever with a Basic license. The default distribution offers more features that are not included in the open source distribution. Compare the Open Source and Basic versions and see for yourself.

Here are instructions for downloading the default distribution of Elasticsearch and Kibana.

Thank you so much Nickpeihl. Will proceed with default distribution as you have suggested. I assume I would just need to uninstall elasticsearch and reintall? Since I've not provisioned anything yet, I've nothing to backup. So should I just do yum remove elasticsearch then proceed with the below link correct? And also the link to the kibana that was provided takes you to 6.8 version whereas elastic search is on 7.6. Will that be ok? btw, having issue with unistalling elasticsearch.

https://www.elastic.co/guide/en/elasticsearch/reference/7.6/rpm.html#rpm-repo

https://www.elastic.co/guide/en/kibana/6.8/rpm.html#rpm-repo

Yes, you will need to uninstall your existing open source distribution of elasticsearch to install the default distribution. Also, sorry about linking the differently versioned documentations. You can use either 7.6 or 6.8, but you must use the same version for both Elasticsearch and Kibana. I recommend installing 7.6 if you want bleeding edge features and easier upgrades in the future. :smiley:

Appreciate your help. I successfully uninstalled kibana and elasticsearch and was able to install both (7.6) version. However, getting error when attempting to start elasticsearch. I rebooted the server as well. Do you have any idea what the issue is? Thanks again.

elasticsearch.yml output.

#cluster.name: graylog
cluster.name: elastic_Server
action.auto_create_index: false

------------------------------------ Node ------------------------------------

Use a descriptive name for the node:

#node.name: node-1

Add custom attributes to the node:

#node.attr.rack: r1

----------------------------------- Paths ------------------------------------

Path to directory where to store the data (separate multiple locations by comma):

path.data: /var/lib/elasticsearch

Path to log files:

path.logs: /var/log/elasticsearch

----------------------------------- Memory -----------------------------------

Lock the memory on startup:

#bootstrap.memory_lock: true

Make sure that the heap size is set to about half the memory available

on the system and that the owner of the process is allowed to use this

limit.

Elasticsearch performs poorly when the system is swapping the memory.

---------------------------------- Network -----------------------------------

Set the bind address to a specific IP (IPv4 or IPv6):

network.host: 10.40.21.3

Set a custom port for HTTP:

http.port: 9200

For more information, consult the network module documentation.

--------------------------------- Discovery ----------------------------------

Pass an initial list of hosts to perform discovery when this node is started:

The default list of hosts is ["127.0.0.1", "[::1]"]

#discovery.seed_hosts: ["host1", "host2"]

Bootstrap the cluster using an initial set of master-eligible nodes:

#cluster.initial_master_nodes: ["node-1", "node-2"]

For more information, consult the discovery and cluster formation module documentation.

---------------------------------- Gateway -----------------------------------

Block initial recovery after a full cluster restart until N nodes are started:

#gateway.recover_after_nodes: 3

For more information, consult the gateway module documentation.

---------------------------------- Various -----------------------------------

Require explicit names when deleting indices:

#action.destructive_requires_name: true

==== ll file directory ========
ll /var/lib/elasticsearch
total 0
drwxr-sr-x. 3 996 992 15 Mar 18 23:39 nodes

==== LOG output ====

[2020-03-20T17:26:24,275][ERROR][o.e.b.Bootstrap ] [greylog] Exception
java.lang.IllegalStateException: failed to obtain node locks, tried [[/var/lib/elasticsearch]] with lock id [0]; maybe these locations are not writable or multiple nodes were started without increasing [node.max_local_storage_nodes] (was [1])?
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:301) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.node.Node.(Node.java:277) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.node.Node.(Node.java:257) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:221) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) [elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) [elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) [elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) [elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) [elasticsearch-cli-7.6.1.jar:7.6.1]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-cli-7.6.1.jar:7.6.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) [elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-7.6.1.jar:7.6.1]
Caused by: java.io.IOException: failed to obtain lock on /var/lib/elasticsearch/nodes/0
at org.elasticsearch.env.NodeEnvironment$NodeLock.(NodeEnvironment.java:223) ~[elasticsearch-7.6.1.jar:7.6.1]
at org.elasticsearch.env.NodeEnvironment.(NodeEnvironment.java:271) ~[elasticsearch-7.6.1.jar:7.6.1]
... 12 more

I changed the permission on /var/lib/elasticsearch and that got it moving. kibana and elasticsearch are up and running. Thank you so much.

1 Like