Kibana Service won't start after 7.8 -> 7.10 Upgrade

Moritz_Kiesewetter · December 16, 2020, 12:58pm

My Cluster:
3x CentOS 7
Elasticsearch 7.10
Logstash 7.10
Kibana 7.10
Filebeat 7.10

So i just upgraded my Cluster to version 7.10
Everything works fine (even Elasticsearch after messing with jvm.options a bit), but Kibana simply won't start. I redid the kibana.yml -> still no progress.

If i check the service after i started it - i can see it working for 5 Seconds - after that it dies - then it's working for 5 seconds - and then it finally dies and stays that way.

If i check with "systemctl status kibana -l" i get the following Error:

Dec 16 13:51:59 xxxxxxx01 kibana[15016]: {"type":"log","@timestamp":"2020-12-16T12:51:59Z","tags":["info","plugins-s ystem"],"pid":15016,"message":"Setting up [99] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,cod e,usageCollection,xpackLegacy,ossTelemetry,telemetryCollectionManager,telemetry,telemetryCollectionXpack,kibanaUsageC ollection,securityOss,newsfeed,mapsLegacy,kibanaLegacy,translations,share,legacyExport,embeddable,uiActionsEnhanced,e xpressions,data,home,observability,cloud,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,man agement,indexPatternManagement,advancedSettings,fileUpload,savedObjects,dashboard,visualizations,visTypeVega,visTypeT imelion,timelion,features,upgradeAssistant,security,snapshotRestore,enterpriseSearch,encryptedSavedObjects,ingestMana ger,indexManagement,remoteClusters,crossClusterReplication,indexLifecycleManagement,dashboardMode,beatsManagement,tra nsform,ingestPipelines,maps,licenseManagement,graph,dataEnhanced,visTypeTable,visTypeMarkdown,tileMap,regionMap,input ControlVis,visualize,esUiShared,charts,lens,visTypeVislib,visTypeTimeseries,rollup,visTypeTagcloud,visTypeMetric,watc her,discover,discoverEnhanced,savedObjectsManagement,spaces,reporting,lists,eventLog,actions,case,alerts,stackAlerts, triggersActionsUi,alerting,alertingBuiltins,ml,securitySolution,infra,monitoring,logstash,apm,uptime,bfetch,canvas]"}
Dec 16 13:51:59 xxxxxxx01 kibana[15016]: {"type":"log","@timestamp":"2020-12-16T12:51:59Z","tags":["fatal","root"]," pid":15016,"message":"TypeError: (0 , _routes.registerMsearchRoute) is not a function\n at SearchService.setup (/u sr/share/kibana/src/plugins/data/server/search/search_service.js:79:38)\n at DataServerPlugin.setup (/usr/share/ki bana/src/plugins/data/server/plugin.js:63:44)\n at PluginWrapper.setup (/usr/share/kibana/src/core/server/plugins/ plugin.js:86:26)\n at PluginsSystem.setupPlugins (/usr/share/kibana/src/core/server/plugins/plugins_system.js:72:2 5)"}
Dec 16 13:51:59 xxxxxx01 kibana[15016]: {"type":"log","@timestamp":"2020-12-16T12:51:59Z","tags":["info","plugins-s ystem"],"pid":15016,"message":"Stopping all plugins."}
Dec 16 13:51:59 xxxxxxx01 kibana[15016]: FATAL TypeError: (0 , _routes.registerMsearchRoute) is not a function

What am i supposed to do? I cannot find any current log in /var/log/kibana...
Thanks in advance!

Moritz

Larry_Gregory · December 16, 2020, 1:52pm

Hey @Moritz_Kiesewetter, sorry to hear you're having trouble with your upgrade.

Can you post your complete kibana.yml file, but redact any sensitive information such as passwords and encryption keys?

Do you have any third-party (custom) plugins installed?

Moritz_Kiesewetter · December 16, 2020, 2:04pm

Hey @Larry_Gregory , thanks for the fast reply!

Here's my Kibana.yml, it's really defaulty, as i said i've redone it after the service started to fail.

[root@xxxxxxxxx01 kibana]# cat kibana.yml
# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "xxxxxx.xxxxxx.xx"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: "/secOps"

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false

# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576

# The Kibana server's name.  This is used for display purposes.
server.name: "xxxxxx01"

# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://X.X.X.X:9200", "http://X.X.X.X:9200", "http://X.X.X.X:9200"]

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "kibana_system"
elasticsearch.password: "*******"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
server.ssl.enabled: true
server.ssl.certificate: /etc/kibana/certs/kibana-certificate.pem
server.ssl.key: /etc/kibana/certs/privateKey.pem


# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/path/to/your/CA.pem" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false

# Specifies the path where Kibana creates the process ID file.
#pid.file: /var/run/kibana.pid

# Enables you to specify a file where Kibana stores log output.
logging.dest: stdout

# Set the value of this setting to true to suppress all logging output.
#logging.silent: false

# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false

# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: false

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000

# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
#i18n.locale: "en"

Also i'm running ElastAlert OS Tool - which (as far a i know) is compatible with Elasticsearch 7.X

Hope that helps.

Moritz

Larry_Gregory · December 16, 2020, 3:07pm

Thanks, that's helpful. Regarding your earlier question:

What am i supposed to do? I cannot find any current log in /var/log/kibana...

Your kibana.yml file is configured to send logs to stdout - I see you're also running this using systemctl, but I'm unfortunately not terribly familiar with that setup. Can you try updating your configuration to output to a file, and see if that writes logs to your expected location? While we're at it, can you enable verbose logging? I'm hoping we can get some more output here to help us track down the issue:

logging.dest: /var/log/kibana/kibana/log
logging.verbose: true

Also i'm running ElastAlert OS Tool - which (as far a i know) is compatible with Elasticsearch 7.X

Thanks - I don't expect this is the cause of your problem, but can you try to remove that plugin, and see if Kibana starts up normally? I want to rule this out as a potential problem before digging in further.

Moritz_Kiesewetter · December 17, 2020, 9:58am

So - I have taken over your suggestions regarding logging into the .yml file.
The directory itself is also created, but no log file is written and so unfortunately I cannot keep any information. Except for the sparse information from /var/log/messages, do they help you?

We have shut down the ElastAlert tool, I cannot delete it - as it is a productive cluster. Even with disbaled ElastAlert service, the Kibana service just won't start.

Thanks in Advance!

Larry_Gregory · December 17, 2020, 2:21pm

So - I have taken over your suggestions regarding logging into the .yml file.
The directory itself is also created, but no log file is written and so unfortunately I cannot keep any information. Except for the sparse information from /var/log/messages, do they help you?

Ah that's a shame . The logs you provided earlier allude to a corrupt installation, but it's hard to say for sure. Can you try to download and extract a fresh copy of 7.10 from our downloads page? Note the default version there is 7.10.1. If you need 7.10.0, then you can find that in the past releases page.

We've seen strange issues in the past where a download/extraction was corrupted for one reason or another, so I'd like to make sure that this isn't the problem here. If possible, try to use the archive version for your environment, rather than the yum/apt package. It's one less moving part to diagnose. If this works, then we can try a fresh packaged version, but we'll take this one step at a time.

We have shut down the ElastAlert tool, I cannot delete it - as it is a productive cluster. Even with disbaled ElastAlert service, the Kibana service just won't start.

Understood - disabling the plugin should be sufficient to rule it out, so thanks for doing that.

Moritz_Kiesewetter · December 17, 2020, 3:20pm

Hi @Larry_Gregory,
we've deleted and reinstalled Kibana completely.
We noticed that there was a Kibana 7.8.1 Installation still on Server.

Now i don't see the error mentioned above anymore, but we get another error:

Dec 17 16:11:28 xxxxxx01 kibana: Error: EISDIR: illegal operation on a directory, open '/var/log/kibana/kibana/log'
After that it turns into Failed state.
This is kinda weird because the directory is tagged the following way:

drwxrwxrwx. 3 kibana kibana 20 Dec 16 16:23 kibana

So actually it should be writeable.

Moritz

Larry_Gregory · December 17, 2020, 3:35pm

Alright, sounds like we're making progress. Do you still have a custom logging.dest set from our earlier experiment? What happens if you remove that configuration and revert to stdout?

Moritz_Kiesewetter · December 18, 2020, 1:42pm

You did it - after switching back to stdout - it now works and the service is up and running!

[root@xxxxxxx01 kibana]# systemctl status kibana
● kibana.service - Kibana
   Loaded: loaded (/etc/systemd/system/kibana.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2020-12-18 14:41:06 CET; 34s ago
 Main PID: 3014 (node)
   CGroup: /system.slice/kibana.service
           └─3014 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli/dist

Just wanted to thank for your time and nerves on this topic.
Really appreciate the work! Thanks a lot.

Larry_Gregory · December 18, 2020, 3:40pm

That's great news! Thanks also for your patience and help with troubleshooting, too.
Enjoy all the new features in 7.10!

system · January 15, 2021, 3:40pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.