Kibana not starting 7.0.1-1

We are trying to upgrade a v6.5 node to the latest (7.0.1-1). Elasticsearch upgraded fine, but Kibana won't start. Kibana was not installed on this node before.

The Kibana logs repeat, showing Kibana is trying. The only error message appears to be this 404:
{"type":"log","@timestamp":"2019-05-20T17:58:46Z","tags":["elasticsearch","query","debug","admin"],"pid":15222,"message":"404\nGET /.kibana/_doc/config%3A7.0.1\n"}

Kibana is configured to use the user "kibana".

I don't see Kibana creating the .kibana index, so I'm not sure when it is supposed to get created. There is no .kibana or .kibana* index yet.

Elasticsearch reports the following from time to time:

[2019-05-20T17:58:56,185][INFO ][o.e.c.m.MetaDataIndexTemplateService] [frost] adding template [.management-beats] for index patterns [.management-beats]

That template is created and looks correct, along with many other templates apparently created by Kibana.

We'd appreciate any ideas.

@L8apex32 can you please post your full log of kibana after the upgrade and your current kibana config?

Also there is any chance you can upgrade first to 6.7.2 before upgrade to 7.0 so you can use the upgrade assistance from 6.7.2 to 7.0 as here is advised here https://www.elastic.co/guide/en/kibana/current/upgrade.html#_upgrading_from_6_x ?

Here is the Kibana config file:

Kibana is served by a back end server. This setting specifies the port to use.

server.port: 5601

Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.

The default is 'localhost', which usually means remote machines will not be able to connect.

To allow connections from remote users, set this parameter to a non-loopback address.

server.host: "frost"

Enables you to specify a path to mount Kibana at if you are running behind a proxy.

Use the server.rewriteBasePath setting to tell Kibana if it should remove the basePath

from requests it receives, and to prevent a deprecation warning at startup.

This setting cannot end in a slash.

#server.basePath: ""

Specifies whether Kibana should rewrite requests that are prefixed with

server.basePath or require that they are rewritten by your reverse proxy.

This setting was effectively always false before Kibana 6.3 and will

default to true starting in Kibana 7.0.

#server.rewriteBasePath: false

The maximum payload size in bytes for incoming server requests.

#server.maxPayloadBytes: 1048576

The Kibana server's name. This is used for display purposes.

server.name: "frost"

The URLs of the Elasticsearch instances to use for all your queries.

elasticsearch.hosts: [ "http://frost:9200" ]

When this setting's value is true Kibana uses the hostname specified in the server.host

setting. When the value of this setting is false, Kibana uses the hostname of the host

that connects to this Kibana instance.

#elasticsearch.preserveHost: true

Kibana uses an index in Elasticsearch to store saved searches, visualizations and

dashboards. Kibana creates a new index if the index doesn't already exist.

#kibana.index: ".thp-kibana"

The default application to load.

#kibana.defaultAppId: "home"

If your Elasticsearch is protected with basic authentication, these settings provide

the username and password that the Kibana server uses to perform maintenance on the Kibana

index at startup. Your Kibana users still need to authenticate with Elasticsearch, which

is proxied through the Kibana server.

elasticsearch.username: "kibana"
elasticsearch.password: "kibana2"

Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.

These settings enable SSL for outgoing requests from the Kibana server to the browser.

server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/certs/asoc.cer
server.ssl.key: /etc/kibana/certs/asoc.key

Optional settings that provide the paths to the PEM-format SSL certificate and key files.

These files validate that your Elasticsearch backend uses the same key files.

#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

Optional setting that enables you to specify a path to the PEM file for the certificate

authority for your Elasticsearch instance.

#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

To disregard the validity of SSL certificates, change this setting's value to 'none'.

elasticsearch.ssl.verificationMode: none

Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of

the elasticsearch.requestTimeout setting.

#elasticsearch.pingTimeout: 1500

Time in milliseconds to wait for responses from the back end or Elasticsearch. This value

must be a positive integer.

elasticsearch.requestTimeout: 60000

List of Kibana client-side headers to send to Elasticsearch. To send no client-side

headers, set this value to (an empty list).

#elasticsearch.requestHeadersWhitelist: [ authorization ]

Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten

by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.

#elasticsearch.customHeaders: {}

Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.

#elasticsearch.shardTimeout: 30000

Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.

#elasticsearch.startupTimeout: 5000

Logs queries sent to Elasticsearch. Requires logging.verbose set to true.

#elasticsearch.logQueries: false

Specifies the path where Kibana creates the process ID file.

#pid.file: /var/run/kibana.pid

Enables you specify a file where Kibana stores log output.

logging.dest: /var/log/kibana.log

Set the value of this setting to true to suppress all logging output.

#logging.silent: false

Set the value of this setting to true to suppress all logging output other than error messages.

#logging.quiet: false

Set the value of this setting to true to log all events, including system usage information

and all requests.

logging.verbose: true

Set the interval in milliseconds to sample system and process performance

metrics. Minimum is 100ms. Defaults to 5000.

#ops.interval: 5000

Specifies locale to be used for all localizable strings, dates and number formats.

#i18n.locale: "en"

new

turn on logging of queries

#xpack.security.encryptionKey: ""

elasticsearch.logQueries: true

xpack Reporting

#xpack.reporting.enabled: "false"
#xpack.reporting.index: ".reporting-7-admin"
#xpack.reporting.encryptionKey: ""

The log file is too large to paste probably. Is there a way to attach a larger file?

It is possible to try the intermediate step to go to 6.7.2. If its needed we'll do it.

@L8apex32 I see you are already using the elasticsearch.hosts which is good.
I would like to see the log output that kibana is creating right after you try to start it.

Also could you check what is the curl response you have when u make the request against the elasticsearch? Also, can you please double check if kibana is running on port 5601 after you started it?

Having the chance to go through 6.7.x before 7.0 would definitely be the best to make sure we won't be missing anything.

Cheers

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.