# Can't upgrade beats / kibana after enabling security

**URL:** https://discuss.elastic.co/t/cant-upgrade-beats-kibana-after-enabling-security/279547
**Category:** Beats
**Tags:** filebeat
**Created:** [July 25, 2021, 5:18am UTC](https://discuss.elastic.co/t/cant-upgrade-beats-kibana-after-enabling-security/279547 "2021-07-25T05:18:55Z")
**Posts on this page:** 5
**Page:** 1

<div class="post-metadata">

### Author: ![DPattee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dpattee/32/37772_2.png) [@DPattee](https://discuss.elastic.co/u/DPattee)
#### Post date: [July 25, 2021, 5:18am UTC](https://discuss.elastic.co/t/cant-upgrade-beats-kibana-after-enabling-security/279547/1 "2021-07-25T05:18:55Z")

</div>

Prior to enabling security on elasticsearch cluster, I was able to easily upgrade beats by just upgrading the beat binary and letting it run.

Since my beat config files (eg filebeat-config.yml) were set with

```auto
setup.template.enabled: true
setup.template.overwrite: true
setup.dashboards.enabled: true
setup.ilm.enabled: auto

```

(as well as the other appropriate settings), the first time a new version of a beat ran all the templates and dashboards would get updated correctly and everything would 'just work'.

After following the tutorials on enabling security, I added the various settings such as

```auto
setup.kibana:
  ssl.enabled
  ssl:
    certificate_authorities: ["${path.config}/elasticsearch-ca.pem"]
    verification_mode: "none"
  username: "foo"
  password: "password"
output.elasticsearch
 (the same, etc) 

```

An existing beat is able to publish its data to the cluster. But upgrading beat fails when it tries to upload the dashboards. The error message in the beat's log, or directly on the console when doing it interactively with the 'setup' command line option, are not super useful. Just 'error 500'

```auto
ndex setup finished.
Loading dashboards (Kibana must be running and reachable)
Exiting: Failed to import dashboard: Failed to load directory /usr/local/Cellar/filebeat-full/7.13.4/libexec/kibana/7/dashboard:
  error loading /usr/local/Cellar/filebeat-full/7.13.4/libexec/kibana/7/dashboard/Filebeat-IBMMQ-Overview.json: returned 500 to import file: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."}
  error loading /usr/local/Cellar/filebeat-full/7.13.4/libexec/kibana/7/dashboard/Filebeat-Iptables-Overview.json: returned 500 to import file: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."}

```

Initially I followed the instructions at [Grant privileges and roles needed for setup | Filebeat Reference [7.13] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/7.13/privileges-to-setup-beats.html) to make a beat setup user, that didn't help. Then I just gave my test user every single option ("\*" for indexes, and all roles possible), that didn't help either.

I'm literally using the 'elastic' user now. Still no dice.

I enabled debug logging on the kibana server with

```auto

logging:
  appenders:
    file:
      type: file
      fileName: /usr/local/var/log/kibana.log
      layout:
        type: pattern
  root:
    appenders: [default, file]
    level: debug

```

And tailing that log, there is a ton of stuff but mainly

```auto
[2021-07-24T22:08:03.813-07:00][DEBUG][http.server.Kibana.cookie-session-storage] Error: Unauthorized
[2021-07-24T22:08:03.813-07:00][DEBUG][http.server.Kibana.cookie-session-storage] Error: Unauthorized
[2021-07-24T22:08:03.813-07:00][DEBUG][http.server.Kibana.cookie-session-storage] Error: Unauthorized

```

Unfortunately at the 'debug' level the tail goes so fast it is hard to see if there is anything else at hte exact time that the client throws the errors, and the on-disk log file grew to over 100 megs in the 10 minutes it took for the beat setup to fail so it is hard to go through. Grepping for the string "error" is primarily the unathorized errors listed above. Grepping for the partial filename of one of the dashboards that failed to upload (eg. "filebeat-cef-network-suspicious") doesn't have any hits.

---

<div class="post-metadata">

### Author: ![DPattee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dpattee/32/37772_2.png) [@DPattee](https://discuss.elastic.co/u/DPattee)
#### Post date: [July 25, 2021, 9:01pm UTC](https://discuss.elastic.co/t/cant-upgrade-beats-kibana-after-enabling-security/279547/2 "2021-07-25T21:01:51Z")

</div>

Further testing this morning with the command line options, shows that the setup --index-management options succeed.

So it seems that it is specifically the dashboard updating that is failing, either automatically (via the options in the config file) or via the setup (with no parameters) line.

If I do index and dashboard separately the seem to work?

Here is an example with metricbeat:

```auto
13:17 Core /usr/local/var/log/metricbeat  sudo /usr/local/opt/metricbeat-full/bin/metricbeat -c my-metricbeat-config.yml --path.config /users/username/github/elastic-configs/metricbeat --strict.perms=false setup
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.

Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Exiting: Failed to import dashboard: Failed to load directory /usr/local/Cellar/metricbeat-full/7.13.4/libexec/kibana/7/dashboard:
  error loading /usr/local/Cellar/metricbeat-full/7.13.4/libexec/kibana/7/dashboard/Metricbeat-Tomcat-overview.json: returned 500 to import file: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."}

13:33 Core /usr/local/var/log/metricbeat  sudo /usr/local/opt/metricbeat-full/bin/metricbeat -c my-metricbeat-config.yml --path.config /users/username/github/elastic-configs/metricbeat --strict.perms=false setup --index-management
Password:
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.

Index setup finished.

13:41 Core /usr/local/var/log/metricbeat  sudo /usr/local/opt/metricbeat-full/bin/metricbeat -c my-metricbeat-config.yml --path.config /users/username/github/elastic-configs/metricbeat --strict.perms=false setup --dashboards
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards

13:54 Core /usr/local/var/log/metricbeat 

```

---

<div class="post-metadata">

### Author: ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)
#### Post date: [July 26, 2021, 11:26am UTC](https://discuss.elastic.co/t/cant-upgrade-beats-kibana-after-enabling-security/279547/3 "2021-07-26T11:26:41Z")

</div>

What version of kibana? What version of filebeat?

---

<div class="post-metadata">

### Author: ![DPattee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dpattee/32/37772_2.png) [@DPattee](https://discuss.elastic.co/u/DPattee)
#### Post date: [July 31, 2021, 8:04pm UTC](https://discuss.elastic.co/t/cant-upgrade-beats-kibana-after-enabling-security/279547/4 "2021-07-31T20:04:25Z")

</div>

I first started activating security back in February and it has been a problem ever sense, so 7.10, 7.12, and 7.13.

I've lost the repro since I finally got them to work on 7.13 by disabling many things randomly, though I expect I'll hit the problem again when 7.14 comes out.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 28, 2021, 10:04pm UTC](https://discuss.elastic.co/t/cant-upgrade-beats-kibana-after-enabling-security/279547/5 "2021-08-28T22:04:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.