Prior to enabling security on elasticsearch cluster, I was able to easily upgrade beats by just upgrading the beat binary and letting it run.
Since my beat config files (eg filebeat-config.yml) were set with
setup.template.enabled: true
setup.template.overwrite: true
setup.dashboards.enabled: true
setup.ilm.enabled: auto
(as well as the other appropriate settings), the first time a new version of a beat ran all the templates and dashboards would get updated correctly and everything would 'just work'.
After following the tutorials on enabling security, I added the various settings such as
setup.kibana:
ssl.enabled
ssl:
certificate_authorities: ["${path.config}/elasticsearch-ca.pem"]
verification_mode: "none"
username: "foo"
password: "password"
output.elasticsearch
(the same, etc)
An existing beat is able to publish its data to the cluster. But upgrading beat fails when it tries to upload the dashboards. The error message in the beat's log, or directly on the console when doing it interactively with the 'setup' command line option, are not super useful. Just 'error 500'
ndex setup finished.
Loading dashboards (Kibana must be running and reachable)
Exiting: Failed to import dashboard: Failed to load directory /usr/local/Cellar/filebeat-full/7.13.4/libexec/kibana/7/dashboard:
error loading /usr/local/Cellar/filebeat-full/7.13.4/libexec/kibana/7/dashboard/Filebeat-IBMMQ-Overview.json: returned 500 to import file: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."}
error loading /usr/local/Cellar/filebeat-full/7.13.4/libexec/kibana/7/dashboard/Filebeat-Iptables-Overview.json: returned 500 to import file: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."}
Initially I followed the instructions at Grant privileges and roles needed for setup | Filebeat Reference [7.13] | Elastic to make a beat setup user, that didn't help. Then I just gave my test user every single option ("*" for indexes, and all roles possible), that didn't help either.
I'm literally using the 'elastic' user now. Still no dice.
I enabled debug logging on the kibana server with
logging:
appenders:
file:
type: file
fileName: /usr/local/var/log/kibana.log
layout:
type: pattern
root:
appenders: [default, file]
level: debug
And tailing that log, there is a ton of stuff but mainly
[2021-07-24T22:08:03.813-07:00][DEBUG][http.server.Kibana.cookie-session-storage] Error: Unauthorized
[2021-07-24T22:08:03.813-07:00][DEBUG][http.server.Kibana.cookie-session-storage] Error: Unauthorized
[2021-07-24T22:08:03.813-07:00][DEBUG][http.server.Kibana.cookie-session-storage] Error: Unauthorized
Unfortunately at the 'debug' level the tail goes so fast it is hard to see if there is anything else at hte exact time that the client throws the errors, and the on-disk log file grew to over 100 megs in the 10 minutes it took for the beat setup to fail so it is hard to go through. Grepping for the string "error" is primarily the unathorized errors listed above. Grepping for the partial filename of one of the dashboards that failed to upload (eg. "filebeat-cef-network-suspicious") doesn't have any hits.