Can't upgrade beats / kibana after enabling security

Prior to enabling security on elasticsearch cluster, I was able to easily upgrade beats by just upgrading the beat binary and letting it run.

Since my beat config files (eg filebeat-config.yml) were set with

setup.template.enabled: true
setup.template.overwrite: true
setup.dashboards.enabled: true
setup.ilm.enabled: auto

(as well as the other appropriate settings), the first time a new version of a beat ran all the templates and dashboards would get updated correctly and everything would 'just work'.

After following the tutorials on enabling security, I added the various settings such as

setup.kibana:
  ssl.enabled
  ssl:
    certificate_authorities: ["${path.config}/elasticsearch-ca.pem"]
    verification_mode: "none"
  username: "foo"
  password: "password"
output.elasticsearch
 (the same, etc) 

An existing beat is able to publish its data to the cluster. But upgrading beat fails when it tries to upload the dashboards. The error message in the beat's log, or directly on the console when doing it interactively with the 'setup' command line option, are not super useful. Just 'error 500'

ndex setup finished.
Loading dashboards (Kibana must be running and reachable)
Exiting: Failed to import dashboard: Failed to load directory /usr/local/Cellar/filebeat-full/7.13.4/libexec/kibana/7/dashboard:
  error loading /usr/local/Cellar/filebeat-full/7.13.4/libexec/kibana/7/dashboard/Filebeat-IBMMQ-Overview.json: returned 500 to import file: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."}
  error loading /usr/local/Cellar/filebeat-full/7.13.4/libexec/kibana/7/dashboard/Filebeat-Iptables-Overview.json: returned 500 to import file: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."}

Initially I followed the instructions at Grant privileges and roles needed for setup | Filebeat Reference [7.13] | Elastic to make a beat setup user, that didn't help. Then I just gave my test user every single option ("*" for indexes, and all roles possible), that didn't help either.

I'm literally using the 'elastic' user now. Still no dice.

I enabled debug logging on the kibana server with


logging:
  appenders:
    file:
      type: file
      fileName: /usr/local/var/log/kibana.log
      layout:
        type: pattern
  root:
    appenders: [default, file]
    level: debug

And tailing that log, there is a ton of stuff but mainly

[2021-07-24T22:08:03.813-07:00][DEBUG][http.server.Kibana.cookie-session-storage] Error: Unauthorized
[2021-07-24T22:08:03.813-07:00][DEBUG][http.server.Kibana.cookie-session-storage] Error: Unauthorized
[2021-07-24T22:08:03.813-07:00][DEBUG][http.server.Kibana.cookie-session-storage] Error: Unauthorized

Unfortunately at the 'debug' level the tail goes so fast it is hard to see if there is anything else at hte exact time that the client throws the errors, and the on-disk log file grew to over 100 megs in the 10 minutes it took for the beat setup to fail so it is hard to go through. Grepping for the string "error" is primarily the unathorized errors listed above. Grepping for the partial filename of one of the dashboards that failed to upload (eg. "filebeat-cef-network-suspicious") doesn't have any hits.

Further testing this morning with the command line options, shows that the setup --index-management options succeed.

So it seems that it is specifically the dashboard updating that is failing, either automatically (via the options in the config file) or via the setup (with no parameters) line.

If I do index and dashboard separately the seem to work?

Here is an example with metricbeat:

13:17  Core  /usr/local/var/log/metricbeat   sudo /usr/local/opt/metricbeat-full/bin/metricbeat -c my-metricbeat-config.yml --path.config /users/username/github/elastic-configs/metricbeat --strict.perms=false setup
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.

Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Exiting: Failed to import dashboard: Failed to load directory /usr/local/Cellar/metricbeat-full/7.13.4/libexec/kibana/7/dashboard:
  error loading /usr/local/Cellar/metricbeat-full/7.13.4/libexec/kibana/7/dashboard/Metricbeat-Tomcat-overview.json: returned 500 to import file: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."}

13:33  Core  /usr/local/var/log/metricbeat   sudo /usr/local/opt/metricbeat-full/bin/metricbeat -c my-metricbeat-config.yml --path.config /users/username/github/elastic-configs/metricbeat --strict.perms=false setup --index-management
Password:
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.

Index setup finished.

13:41  Core  /usr/local/var/log/metricbeat   sudo /usr/local/opt/metricbeat-full/bin/metricbeat -c my-metricbeat-config.yml --path.config /users/username/github/elastic-configs/metricbeat --strict.perms=false setup --dashboards
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards

13:54  Core  /usr/local/var/log/metricbeat  

What version of kibana? What version of filebeat?

I first started activating security back in February and it has been a problem ever sense, so 7.10, 7.12, and 7.13.

I've lost the repro since I finally got them to work on 7.13 by disabling many things randomly, though I expect I'll hit the problem again when 7.14 comes out.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.