Kibana 7.6.0 not starting because cannot install APM plugin

Elastic Stack Kibana
1

Hello. I'm working on upgrading our Elastic setup to version 7.6.0 and I'm having issues with Kibana startup. What I see in the logs is that Kibana is not able to install the "apm" plugin within the default 30 seconds (see logs below).

Here are the logs:

Feb 18 14:15:48 systemd[1]: Started Kibana.
Feb 18 14:15:52 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:15:52Z","tags":["info","plugins-service"],"pid":8900,"message":"Plugin \"case\" is disabled."}
Feb 18 14:15:56 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:15:56Z","tags":["info","plugins-system"],"pid":8900,"message":"Setting up [37] plugins: [usageCollection,metrics,
Feb 18 14:15:56 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:15:56Z","tags":["info","plugins","usageCollection"],"pid":8900,"message":"Setting up plugin"}
Feb 18 14:15:56 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:15:56Z","tags":["info","plugins","metrics"],"pid":8900,"message":"Setting up plugin"}
Feb 18 14:15:56 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:15:56Z","tags":["info","plugins","canvas"],"pid":8900,"message":"Setting up plugin"}
Feb 18 14:15:56 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:15:56Z","tags":["info","plugins","timelion"],"pid":8900,"message":"Setting up plugin"}
Feb 18 14:15:56 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:15:56Z","tags":["info","plugins","features"],"pid":8900,"message":"Setting up plugin"}
Feb 18 14:15:56 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:15:56Z","tags":["info","plugins","apm_oss"],"pid":8900,"message":"Setting up plugin"}
Feb 18 14:15:56 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:15:56Z","tags":["info","plugins","taskManager"],"pid":8900,"message":"Setting up plugin"}
Feb 18 14:15:56 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:15:56Z","tags":["info","plugins","siem"],"pid":8900,"message":"Setting up plugin"}
Feb 18 14:15:56 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:15:56Z","tags":["info","plugins","licensing"],"pid":8900,"message":"Setting up plugin"}
Feb 18 14:15:56 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:15:56Z","tags":["info","plugins","security"],"pid":8900,"message":"Setting up plugin"}
Feb 18 14:15:56 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:15:56Z","tags":["warning","plugins","security","config"],"pid":8900,"message":"Generating a random key for xpack.
Feb 18 14:15:56 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:15:56Z","tags":["warning","plugins","security","config"],"pid":8900,"message":"Session cookies will be transmitte
Feb 18 14:15:56 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:15:56Z","tags":["info","plugins","infra"],"pid":8900,"message":"Setting up plugin"}
Feb 18 14:15:56 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:15:56Z","tags":["info","plugins","encryptedSavedObjects"],"pid":8900,"message":"Setting up plugin"}
Feb 18 14:15:56 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:15:56Z","tags":["warning","plugins","encryptedSavedObjects","config"],"pid":8900,"message":"Generating a random k
Feb 18 14:15:56 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:15:56Z","tags":["info","plugins","code"],"pid":8900,"message":"Setting up plugin"}
Feb 18 14:15:56 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:15:56Z","tags":["info","plugins","data"],"pid":8900,"message":"Setting up plugin"}
Feb 18 14:15:56 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:15:56Z","tags":["info","plugins","share"],"pid":8900,"message":"Setting up plugin"}
Feb 18 14:15:56 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:15:56Z","tags":["info","plugins","home"],"pid":8900,"message":"Setting up plugin"}
Feb 18 14:15:56 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:15:56Z","tags":["info","plugins","spaces"],"pid":8900,"message":"Setting up plugin"}
Feb 18 14:15:56 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:15:56Z","tags":["info","plugins","cloud"],"pid":8900,"message":"Setting up plugin"}
Feb 18 14:15:56 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:15:56Z","tags":["info","plugins","apm"],"pid":8900,"message":"Setting up plugin"}
Feb 18 14:16:26 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:16:26Z","tags":["fatal","root"],"pid":8900,"message":"Error: Setup lifecycle of \"apm\" plugin wasn't completed i
Feb 18 14:16:26 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:16:26Z","tags":["info","plugins-system"],"pid":8900,"message":"Stopping all plugins."}
Feb 18 14:16:26 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:16:26Z","tags":["info","plugins","cloud"],"pid":8900,"message":"Stopping plugin"}
Feb 18 14:16:26 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:16:26Z","tags":["info","plugins","spaces"],"pid":8900,"message":"Stopping plugin"}
Feb 18 14:16:26 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:16:26Z","tags":["info","plugins","home"],"pid":8900,"message":"Stopping plugin"}
Feb 18 14:16:26 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:16:26Z","tags":["info","plugins","share"],"pid":8900,"message":"Stopping plugin"}
Feb 18 14:16:26 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:16:26Z","tags":["info","plugins","data"],"pid":8900,"message":"Stopping plugin"}
Feb 18 14:16:26 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:16:26Z","tags":["info","plugins","code"],"pid":8900,"message":"Stopping plugin"}
Feb 18 14:16:26 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:16:26Z","tags":["info","plugins","encryptedSavedObjects"],"pid":8900,"message":"Stopping plugin"}
Feb 18 14:16:26 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:16:26Z","tags":["info","plugins","infra"],"pid":8900,"message":"Stopping plugin"}
Feb 18 14:16:26 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:16:26Z","tags":["info","plugins","security"],"pid":8900,"message":"Stopping plugin"}
Feb 18 14:16:26 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:16:26Z","tags":["info","plugins","licensing"],"pid":8900,"message":"Stopping plugin"}
Feb 18 14:16:26 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:16:26Z","tags":["info","plugins","siem"],"pid":8900,"message":"Stopping plugin"}
Feb 18 14:16:26 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:16:26Z","tags":["info","plugins","taskManager"],"pid":8900,"message":"Stopping plugin"}
Feb 18 14:16:26 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:16:26Z","tags":["info","plugins","apm_oss"],"pid":8900,"message":"Stopping plugin"}
Feb 18 14:16:26 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:16:26Z","tags":["info","plugins","features"],"pid":8900,"message":"Stopping plugin"}
Feb 18 14:16:26 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:16:26Z","tags":["info","plugins","timelion"],"pid":8900,"message":"Stopping plugin"}
Feb 18 14:16:26 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:16:26Z","tags":["info","plugins","canvas"],"pid":8900,"message":"Stopping plugin"}
Feb 18 14:16:26 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:16:26Z","tags":["info","plugins","metrics"],"pid":8900,"message":"Stopping plugin"}
Feb 18 14:16:26 kibana[8900]: {"type":"log","@timestamp":"2020-02-18T14:16:26Z","tags":["info","plugins","usageCollection"],"pid":8900,"message":"Stopping plugin"}
Feb 18 14:16:26 kibana[8900]:  FATAL  Error: Setup lifecycle of "apm" plugin wasn't completed in 30sec. Consider disabling the plugin and re-start.
Feb 18 14:16:27 systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Feb 18 14:16:27 systemd[1]: kibana.service: Failed with result 'exit-code'.

I've also tried to install the plugin manually and increasing the time available for the installation using the following command but the URL that is used by default does not contain the plugin.

$ /usr/share/kibana# bin/kibana-plugin install apm 2m --allow-root
Attempting to transfer from apm
Attempting to transfer from https://artifacts.elastic.co/downloads/kibana-plugins/apm/apm-7.6.0.zip
Plugin installation was unsuccessful due to error "No valid url specified."

If I do a curl request to that URL I get a 404 response code:

$ curl -I https://artifacts.elastic.co/downloads/kibana-plugins/apm/apm-7.6.x.zip
HTTP/2 404
retry-after: 0
content-type: text/html
accept-ranges: bytes
accept-ranges: bytes
date: Tue, 18 Feb 2020 14:25:12 GMT
via: 1.1 varnish
x-served-by: cache-ams21042-AMS
x-cache: MISS
x-cache-hits: 0
x-timer: S1582035912.281037,VS0,VE372
server: ElasticInfrastructure
content-length: 78

I also tried to disable the plugin in the "kibana.yml" file using the config option "xpack.apm.enabled" but because of this bug it does not work.

Could someone advice on how to proceed here without downgrading to version 7.5.2?

Thank you very much for your help.
Regards
/José Carlos

2

Hello José, Thank you for reaching out.

First, let's make sure the kibana user (elasticsearch.username and elasticsearch.password in kibana.yml) has sufficient permissions to create the required index (.apm-agent-configuration) in APM startup. Use the following API against the elasticsearch host:

curl -u 'username:password' http://es-host:9200/_security/_authenticate

Check to make sure kibana.roles includes "kibana_system" in the list. Then call this API:

curl -u 'username:password' -H 'Content-Type: application/json' -d'{"index":[{"names":[".apm-agent-configuration"],"privileges":["all"]}]}' http://es-host:9200/_security/user/_has_privileges

Check to make sure you see this in the response:

"index" : {".apm-agent-configuration" : {"all" : true}

If both of these checks pass, then there may be some other error preventing kibana from creating or accessing the .apm-agent-configuration index, which is crashing the APM plugin. In kibana.yml, you can set logging.verbose: true and paste it here to give us a better idea of what the problem could be.

3

Hi Oliver. Thanks for checking.

I'm a bit lost now. I don't have security enabled in Kibana (neither in Elasticsearch) so the requests you asked me to run are failing.

Before the upgrade to Elastic 7.6.0 I also had APM running (at least the UI) and I don't see any .apm-agent-configuration index.

~# curl -X GET "localhost:9200/.apm-agent-configuration?pretty"
{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [.apm-agent-configuration]",
        "index_uuid" : "_na_",
        "resource.type" : "index_or_alias",
        "resource.id" : ".apm-agent-configuration",
        "index" : ".apm-agent-configuration"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index [.apm-agent-configuration]",
    "index_uuid" : "_na_",
    "resource.type" : "index_or_alias",
    "resource.id" : ".apm-agent-configuration",
    "index" : ".apm-agent-configuration"
  },
  "status" : 404
}

Maybe I was not very clear in my first post. I'm running a BASIC license and I'm not using APM at all. Kibana 7.4.2 was running perfectly (APM plugin - at least the UI) loaded fine but when installing Kibana 7.6.0 it fails.

I'm perfectly fine with disable it but as I mentioned previously I was not able to do it because of an existing bug.

This is my kibana.yml config:

# change defaults
server.host: 0.0.0.0
server.port: 5601

# security config
## https://www.elastic.co/guide/en/kibana/current/production.html#csp-strict-mode
csp.strict: true

## SSL config disabled because it's handled by the OAuth proxy
## https://www.elastic.co/guide/en/kibana/current/configuring-tls.html
server.ssl.enabled: false

# Elasticsearch config
elasticsearch.hosts:
  - http://XX.XX.XX.XX:9200
  - http://XX.XX.XX.XX:9200
  - http://XX.XX.XX.XX:9200

# Kibana config
logging.json: true

The following are the Kibana debug logs (they are cut out because of message size limitation):

Feb 19 10:32:08 systemd[1]: Started Kibana.
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:11Z","tags":["debug","config"],"pid":14625,"message":"Marking config path as handled: server"}
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:11Z","tags":["debug","config"],"pid":14625,"message":"Marking config path as handled: csp"}
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:11Z","tags":["debug","config"],"pid":14625,"message":"Marking config path as handled: plugins"}
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:11Z","tags":["debug","config"],"pid":14625,"message":"Marking config path as handled: dev"}
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:11Z","tags":["debug","config"],"pid":14625,"message":"Marking config path as handled: server"}
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:11Z","tags":["debug","config"],"pid":14625,"message":"Marking config path as handled: csp"}
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:11Z","tags":["debug","config"],"pid":14625,"message":"Marking config path as handled: elasticsearch"}
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:11Z","tags":["debug","config"],"pid":14625,"message":"Marking config path as handled: uiSettings"}
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:11Z","tags":["debug","config"],"pid":14625,"message":"Marking config path as handled: path"}
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:13Z","tags":["debug","config"],"pid":14625,"message":"Marking config path as handled: csp"}
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:13Z","tags":["debug","config"],"pid":14625,"message":"Marking config path as handled: elasticsearch"}
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:13Z","tags":["debug","config"],"pid":14625,"message":"Marking config path as handled: logging"}
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:13Z","tags":["debug","config"],"pid":14625,"message":"Marking config path as handled: server"}
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:13Z","tags":["debug","config"],"pid":14625,"message":"Marking config path as handled: plugins"}
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:13Z","tags":["debug","config"],"pid":14625,"message":"Marking config path as handled: dev"}
...
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:13Z","tags":["debug","root"],"pid":14625,"message":"setting up root"}
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:13Z","tags":["debug","server"],"pid":14625,"message":"setting up server"}
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:13Z","tags":["debug","plugins-service"],"pid":14625,"message":"Discovering plugins"}
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:13Z","tags":["debug","plugins-discovery"],"pid":14625,"message":"Discovering plugins..."}
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:13Z","tags":["debug","plugins-discovery"],"pid":14625,"message":"Scanning \"/usr/share/kibana/src/plugins\" f
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:13Z","tags":["debug","plugins-discovery"],"pid":14625,"message":"Scanning \"/usr/share/kibana/x-pack/plugins\
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:13Z","tags":["debug","plugins-discovery"],"pid":14625,"message":"Scanning \"/usr/share/kibana/plugins\" for p
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:13Z","tags":["debug","plugins-discovery"],"pid":14625,"message":"Scanning \"/usr/share/kibana-extra\" for plu
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:13Z","tags":["debug","plugins-discovery"],"pid":14625,"message":"Successfully discovered plugin \"apm_oss\" a
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:13Z","tags":["debug","config"],"pid":14625,"message":"Marking config path as handled: kibana"}
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:13Z","tags":["debug","config"],"pid":14625,"message":"Marking config path as handled: elasticsearch"}
Feb 19 10:32:13 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:13Z","tags":["debug","config"],"pid":14625,"message":"Marking config path as handled: path"}
...
Feb 19 10:32:18 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:18Z","tags":["info","plugins","spaces"],"pid":14625,"message":"Setting up plugin"}
Feb 19 10:32:18 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:18Z","tags":["debug","plugins-system"],"pid":14625,"message":"Setting up plugin \"cloud\"..."}
Feb 19 10:32:18 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:18Z","tags":["debug","plugins","cloud"],"pid":14625,"message":"Initializing plugin"}
Feb 19 10:32:18 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:18Z","tags":["debug","config"],"pid":14625,"message":"Marking config path as handled: xpack,cloud"}
Feb 19 10:32:18 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:18Z","tags":["info","plugins","cloud"],"pid":14625,"message":"Setting up plugin"}
Feb 19 10:32:18 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:18Z","tags":["debug","plugins","cloud"],"pid":14625,"message":"Setting up Cloud plugin"}
Feb 19 10:32:18 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:18Z","tags":["debug","plugins-system"],"pid":14625,"message":"Setting up plugin \"apm\"..."}
Feb 19 10:32:18 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:18Z","tags":["debug","plugins","apm"],"pid":14625,"message":"Initializing plugin"}
Feb 19 10:32:18 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:18Z","tags":["info","plugins","apm"],"pid":14625,"message":"Setting up plugin"}
Feb 19 10:32:18 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:18Z","tags":["debug","config"],"pid":14625,"message":"Marking config path as handled: xpack,apm"}
Feb 19 10:32:48 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:48Z","tags":["debug","root"],"pid":14625,"message":"shutting root down"}
Feb 19 10:32:48 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:48Z","tags":["fatal","root"],"pid":14625,"message":"Error: Setup lifecycle of \"apm\" plugin wasn't completed
Feb 19 10:32:48 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:48Z","tags":["debug","server"],"pid":14625,"message":"stopping server"}
Feb 19 10:32:48 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:48Z","tags":["debug","legacy-service"],"pid":14625,"message":"stopping legacy service"}
Feb 19 10:32:48 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:48Z","tags":["debug","plugins-service"],"pid":14625,"message":"Stopping plugins service"}
Feb 19 10:32:48 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:48Z","tags":["info","plugins-system"],"pid":14625,"message":"Stopping all plugins."}
Feb 19 10:32:48 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:48Z","tags":["debug","plugins-system"],"pid":14625,"message":"Stopping plugin \"cloud\"..."}
Feb 19 10:32:48 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:48Z","tags":
...
Feb 19 10:32:48 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:48Z","tags":["debug","plugins-system"],"pid":14625,"message":"Stopping plugin \"infra\"..."}
Feb 19 10:32:48 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:48Z","tags":["info","plugins","infra"],"pid":14625,"message":"Stopping plugin"}
Feb 19 10:32:48 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:48Z","tags":["debug","plugins-system"],"pid":14625,"message":"Stopping plugin \"licensing\"..."}
Feb 19 10:32:48 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:48Z","tags":["info","plugins","licensing"],"pid":14625,"message":"Stopping plugin"}
Feb 19 10:32:48 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:48Z","tags":["debug","plugins-system"],"pid":14625,"message":"Stopping plugin \"siem\"..."}
Feb 19 10:32:48 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:48Z","tags":["info","plugins","siem"],"pid":14625,"message":"Stopping plugin"}
Feb 19 10:32:48 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:48Z","tags":["debug","plugins","siem"],"pid":14625,"message":"plugin stopped"}
Feb 19 10:32:48 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:48Z","tags":["debug","plugins-system"],"pid":14625,"message":"Stopping plugin \"taskManager\"..."}
Feb 19 10:32:48 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:48Z","tags":["info","plugins","taskManager"],"pid":14625,"message":"Stopping plugin"}
Feb 19 10:32:48 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:48Z","tags":["debug","elasticsearch-service"],"pid":14625,"message":"Stopping elasticsearch service"}
Feb 19 10:32:48 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:48Z","tags":["debug","elasticsearch-service"],"pid":14625,"message":"Closing elasticsearch clients"}
Feb 19 10:32:48 kibana[14625]: {"type":"log","@timestamp":"2020-02-19T10:32:48Z","tags":["debug","http","server","Kibana"],"pid":14625,"message":"stopping http server"}
Feb 19 10:32:48 kibana[14625]:  FATAL  Error: Setup lifecycle of "apm" plugin wasn't completed in 30sec. Consider disabling the plugin and re-start.
Feb 19 10:32:50 systemd[1]: kibana.service: Main process exited, code=exited, status=1/FAILURE
Feb 19 10:32:50 systemd[1]: kibana.service: Failed with result 'exit-code'.

I would appreciate some advice on how to proceed. Thank you very much.

Regards,
/José Carlos

4

i had the same problem today, turned out that elastic was failing the masternode discovery.
try this call: http://localhost:9200/_xpack/license
If you get a response error saying something like "master_discovery_exception" than your elasticsearch.yml has wrong values on the discovery section.

if so you need to change these values in your elasticsearch.yml:
discovery.seed_hosts: ["127.0.0.1"]
cluster.initial_master_nodes: ["local-node"]

put in the first one the ip of you node and in the second one the name of you master node; if you are running a single node cluster you need to put the node name of the current node which is defined in the same file as "node.name"

After changing this, restart elasticsearch and try again the first call, the correct output should be like this:
{
"license" : {
"status" : "active",
"uid" : "1b99d85e-9553-40ae-8714-70272ad9bec6",
"type" : "basic",
"issue_date" : "2020-02-20T10:53:04.881Z",
"issue_date_in_millis" : 1582195984881,
"max_nodes" : 1000,
"issued_to" : "local-cluster",
"issuer" : "elasticsearch",
"start_date_in_millis" : -1
}
}

if so, start kibana and everything should work.

2 Likes
5

Hi Andrea. Thank you for checking this. Unfortunately, my issue is not the same as you faced.

When I run http://localhost:9200/_xpack/license I get a proper response and no exception about master_discovery_exception.

Elasticsearch cluster is running fine:

~# curl -X GET localhost:9200/_cluster/health?pretty
{
  "cluster_name" : "a-cluster",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 18,
  "active_shards" : 36,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}
6

I am having the exact same issue as @josejfb with Kibana 7.6.0. I have tried to disable the APM plugin following the doc here but this does not work either. Any suggestion on how to sort it out is appreciated.

7

I too am struggling with the same problem. Kibana 7.6.0 service fails to start with the following error:

"FATAL Error: Setup lifecycle of "apm" plugin wasn't completed in 30sec. Consider disabling the plugin and re-start."

Attempting to install the apm plugin by increasing the timeout is of no help as the "apm-7.6.0.zip" file does not appear to exist as a downloadable plugin:

bin/kibana-plugin install --timeout 5m apm --allow-root
Attempting to transfer from apm
Attempting to transfer from https://artifacts.elastic.co/downloads/kibana-plugins/apm/apm-7.6.0.zip
Plugin installation was unsuccessful due to error "No valid url specified."

Where can we get the apm-7.6.0.zip plugin?

Thanks!

8

Getting the same behaviour/problems on Kibana startup for 7.6.0:

{"type":"log","@timestamp":"2020-02-25T16:06:45Z","tags":["info","plugins-service"],"pid":30297,"message":"Plugin "case" is disabled."}
{"type":"log","@timestamp":"2020-02-25T16:06:45Z","tags":["info","plugins-service"],"pid":30297,"message":"Plugin "security" is disabled."}

Kibana started but not able to load/browse, and getting SSL errors:

{"type":"error","@timestamp":"2020-02-25T16:27:47Z","tags":["connection","client","error"],"pid":30297,"level":"error","error":{"message":"140707852973888:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n","name":"Error","stack":"Error: 140707852973888:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"},"message":"140707852973888:error:1408F09C:SSL routines:ssl3_get_record:http request:../deps/openssl/openssl/ssl/record/ssl3_record.c:322:\n"}

The configuration hasn't change, it was working fine under previous version 7.5.1

9

Error: Setup lifecycle of "apm" plugin wasn't completed in 30sec. Consider disabling the plugin and re-start.

When Kibana starts a plugin it goes through several lifecycles, setup, start and finally stop. This error indicates that Kibana was unable to "setup" the APM plugin. The only way to get past this error is to disable the plugin (not possible because of the bug) or get the plugin to start working.

Getting the plugin to work

Can you paste your elasticsearch logs from when Kibana starts up?

Other ways to disable the APM plugin

  1. You mentioned that you're not using security from basic, if you're not using any other basic features you could try the OSS version of Kibana as that won't come with the x-pack/apm plugin pre-installed.
  2. Remove the plugin from disk rm -rf ./kibana/x-pack/plugins/apm
  3. Patch your local Kibana with the patch from [APM] Add `xpack.apm.enabled` key to config schema by dgieselaar · Pull Request #57539 · elastic/kibana · GitHub and disable it via configuration.
10

I wanted to thank you as I just tried your solution. I changed the values in discovery.seed and cluster.initial_master_nodes - restarted kibana and it came up. Many thanks

11

Hi @rudolf. Thanks for taking a look at this. Your comment "Can you paste your elasticsearch logs from when Kibana starts up?" pointed me in the right direction on where to look for the problem.

Basically, the connection between the Kibana instance and the Elasticsearch cluster instances was not working. Once the connection was restored Kibana was able to install the APM plugin.

I mark topic as solved and want to thank you for your help.

Regards,
/José Carlos

12

Glad I could help! Can you share more about what the actual problem was? Why was Kibana's connection not working? When we start Kibana we check that Elasticsearch is up by connecting, only if that succeeds we start loading plugins. So I want to be sure we're not somehow starting plugins without being sure that Elasticsearch is up.

What we don't check, however, is whether Elasticsearch is green, so if Elasticsearch is incorrectly configured but still accepting connections we could start the APM plugin.

13

Hi @rudolf. I was doing some refactor work on the infrastructure code and I've missed the security rule to enable the connection between the Kibana VM and the Elasticsearch cluster VMs, that's the reason why the connection was not working.

For what I saw in the logs, it seems that some of the Kibana plugins don't need Elasticsearch up to set them up...Fyi, I've checked the Elasticsearch logs while starting Kibana and only saw entries regarding the APM plugin.

Let me know if you need more information. Thank you again for your help.

/José Carlos

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.