Upgrade to 7.6: Stack Monitoring Only With Security Enabled Possible?

tomx1 · February 25, 2020, 12:13pm

Hey,

since upgrading our cluster to 7.6.0 I'm facing the problem, that it is not possible anymore to use the "Stack Monitoring" App in Kibana. The following error message appears:

{
  "type": "error",
  "@timestamp": "2020-02-25T12:02:44Z",
  "tags": [
    "error",
    "monitoring"
  ],
  "pid": 6,
  "level": "error",
  "error": {
    "message": "[invalid_index_name_exception] Invalid index name [_security], must not start with _, -, or +, with { index_uuid=\"_na_\" & index=\"_security\" }",
    "name": "Error",
    "stack": "Error: [invalid_index_name_exception] Invalid index name [_security], must not start with _, -, or +, with { index_uuid=\"_na_\" & index=\"_security\" }\n    at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n    at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n    at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n    at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n    at IncomingMessage.emit (events.js:203:15)\n    at endReadableNT (_stream_readable.js:1143:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"
  },
  "message": "[invalid_index_name_exception] Invalid index name [_security], must not start with _, -, or +, with { index_uuid=\"_na_\" & index=\"_security\" }"
}

I'm collecting monitoring data from our 10 node production cluster nodes with metricbeat and we push it to a standalone single node "cluster" which has no xpack security enabled. Our 10 node production cluster is using xpack security.
After upgrading everything to 7.6.0 it seems that enabling xpack security on the monitoring node is mandatory, but I did not find any hint about that in the release notes?

However, if I enable security at the monitoring node and create a user which exists at the production AND monitoring cluster, the Stack Monitoring App in Kibana is working again?

joshdover · February 25, 2020, 8:14pm

Hi there!

Just a few questions for you to help me figure out what is going on here:

Just to confirm your setup:
- 10 node production cluster w/ security + monitoring enabled (send to separate cluster)
- 1 node monitoring cluster w/ no security
The error message you are getting, you receive that in the production cluster?

Can you try setting these settings in your kibana.yml? I suspect we may have a bug similar to this one where we are not merging configurations correctly.

xpack.monitoring.elasticsearch.username: null
xpack.monitoring.elasticsearch.password: null

tomx1 · February 26, 2020, 11:53am

10 node production cluster w/ security + monitoring enabled (send to separate cluster)

Correct. Monitoring stats were collected via a dedicated metricbeat instance and pushed to the monitoring cluster

1 node monitoring cluster w/ no security

Yes.

The error message you are getting, you receive that in the production cluster?

The error message is generated by kibana which is logging to syslog

Can you try setting these settings in your kibana.yml? I suspect we may have a bug similar to this one where we are not merging configurations correctly.

I've tried that, unfortunately without success. I'm receiving the same error message as above. I may have to mention that the hole setup is running under docker swarm, maybe that makes a difference for 7.6?

So to describe the Docker setup in short.

5 baremetal server each running:

1 Docker Elasticsearch Hot Node (SSD Storage)
1 Docker Elasticsearch Warm Node (SAS Storage)
1 Docker Metricbeat Instance collecting monitoring data from hot and warm nodes and pushes it to monitoring cluster

Somewhere inside the Docker Swarm cluster the following containers are running (replicated mode):

1 Docker Kibana instance
1 Docker Metricbeat instance collect Kibana monitoring stats

1 Virtual Machine (Part of Swarm Cluster and running on separate hardware):

1 Docker Elasticsearch Monitoring Cluster

chrisronline · February 26, 2020, 6:01pm

Hi @tomx1,

This is a known issue: https://github.com/elastic/kibana/issues/53129

We have a fix in the works now: https://github.com/elastic/kibana/pull/53306

tomx1 · February 27, 2020, 8:26am

Are you sure that we are talking about the same issue here? I'm not talking about "how to setup Stack Monitoring", it's about that Stack Monitoring is not working if xpack.security.enabled is set to true on the monitoring node.

Here are two screenshots...

1.) xpack.security.enabled = true on Monitoring Node