Beats setup fails with custom kibana_index

Elastic Stack Beats
1

When setting up a new secure stack I noticed that beat setup fails when all of these are true:

I first noticed this issue on 7.3.1, but I was able to reproduce it on 7.4.0.
It happens at least on auditbeat and metricbeat.

Reproduced with following changes to standard config:

elasticsearch.yml

xpack.security.enabled: true

kibana.yml

server.host: "0.0.0.0"
elasticsearch.username: "kibana"
elasticsearch.password: "kibana123"
kibana.index: ".kibana-foo"

metricbeat setup.yml

metricbeat:
  config:
    modules:
      path: "${path.config}/modules.d/*.yml"
      reload:
        enabled: false
output:
  elasticsearch:
    enabled: true
    hosts: ["127.0.0.1:9200"]
    protocol: "http"
    username: "metricbeat_setup_user"
    password: "metricbeat_setup_user123"
    ssl.enabled: false
  logstash:
    enabled: false

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

setup:
  ilm:
    check_exists: false
    enabled: false
    overwrite: false
  kibana:
    host: "127.0.0.1"
    protocol: "http"
    ssl.enabled: false
    username: "metricbeat_setup_user"
    password: "metricbeat_setup_user123"
  dashboards:
    enabled: true
    kibana_index: ".kibana-foo"
  template:
    enabled: true
    name: "metricbeat-bar-7.4.0"
    overwrite: true
    pattern: "metricbeat-bar-7.4.0-*"
    settings:
      index:
        codec: best_compression
        number_of_shards: 1

Setup error

~/metricbeat-7.4.0-linux-x86_64> ./metricbeat setup -c setup.yml -v
ILM policy and write alias loading not enabled.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Exiting: Failed to import dashboard: Failed to load directory /home/vagrant/metricbeat-7.4.0-linux-x86_64/kibana/7/dashboard:
  error loading /home/vagrant/metricbeat-7.4.0-linux-x86_64/kibana/7/dashboard/Metricbeat-aerospike-overview.json: returned 403 to import file: <nil>. Response: {"statusCode":403,"error":"Forbidden","message":"Unable to bulk_create dashboard,visualization"}
  error loading /home/vagrant/metricbeat-7.4.0-linux-x86_64/kibana/7/dashboard/Metricbeat-apache-overview.json: returned 403 to import file: <nil>. Response: {"statusCode":403,"error":"Forbidden","message":"Unable to bulk_create dashboard,search,visualization"}
  (...)

Role/user details:

http://127.0.0.1:9200/_security/role/metricbeat_setup?pretty

{
"metricbeat_setup" : {
"cluster" : [
"monitor",
"manage_ilm",
"manage_ml"
],
"indices" : [
{
"names" : [
"metricbeat-*"
],
"privileges" : [
"manage",
"read"
],
"allow_restricted_indices" : false
}
],
"applications" : [ ],
"run_as" : [ ],
"metadata" : { },
"transient_metadata" : {
"enabled" : true
}
}
}

http://127.0.0.1:9200/_security/user/metricbeat_setup_user?pretty

{
"metricbeat_setup_user" : {
"username" : "metricbeat_setup_user",
"roles" : [
"metricbeat_setup",
"kibana_user",
"ingest_admin",
"beats_admin"
],
"full_name" : "",
"email" : "",
"metadata" : { },
"enabled" : true
}
}

It works when

  • I comment out kibana_index: ".kibana-foo" in metric and kibana
    or
  • I run with custom index but on elastic user (instead of metricbeat_setup_user)

Should something else be added to metricbeat_setup_user or metricbeat_setup role when dealing with non standard kibana-index?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.