Upgrade stopped monitoring

Elastic Stack Elasticsearch
1

Environment Info
We had two clusters with version 6.2.1, where one is dedicated for monitoring. The monitoring cluster was also monitoring logstash (Docker 6.2). We upgraded the monitoring cluster to 6.3 from 6.2.1.

Result

  • ES cluster that is 6.2.1 is still sending data to monitoring without any issues
  • Logstash is not sending any monitoring data
  • ES monitoring cluster 6.3 (Local Monitoring) is also not working.

Looking at _cat/indices in monitoring cluster (6.3), there were no new indices created for logstash (monitoring-logstash-6-2018.06....). It stopped creating indices that would host logstash data, which rolls daily

Also, querying GET /.monitoring-es-6-2018.06.*/_search doesn't yield any result for local (monitoring) cluster post the time of upgrade.

Validated the below settings and ensured they are proper w.r.t ES URL, user/password etc.

  1. License It is valid in both monitoring and the other es cluster
  2. xPack settings with appropriate local exporter (for monitoring)
  3. xpack configuration in logstash

Not sure how the 6.2.1 cluster alone sends monitoring data to 6.3, but not the rest (logstash and local monitoring).

Am I missing anything? Any advice would be helpful.

2

Trying

https://www.elastic.co/guide/en/elasticsearch/reference/6.3/monitoring-settings.html....

"To enable data collection, you must also set xpack.monitoring.collection.enabled to true. Its default value is false."

Will respond with status...

Now it started monitoring... But shows below message when kibana is used to see the data

Your Invalid license is inactive
Your license will expire on June 29, 2018 12:24 AM UTC. + .....

But GET _xpack/license provides below

"expiry_date": "2019-..... with status active
Should I register license again after enabling monitoring?

Hmm. Registering again didn't yield right result. The messages are same in kibana
Navigation Path
Kibana-Monitoring-<<Lists clusters both 6.2.1 and 6.3 >>

6.3 says license invalid with June 29, though the GET _xpack/license API in the cluster gives me a different day, which is very far and not basic.

Again - Am I missing any :frowning:

3

Still the issue persist. The licenses are all good and valid for long time.

Not able to view the data from kibana monitoring, though I see the data get collected by querying indices.
:frowning:

4

I have kibana working great but not logstash. Want to trade configs??

5

I just executed below against my new 6.3 monitoring cluster, and it started monitoring logstash, and my monitoring elastic environment. Even before I executed this command, the other elastic environment, which was in 6.2 continued to send data to the monitoring cluster without executing the command.

PUT _cluster/settings
{
"persistent": {
"xpack.monitoring.collection.enabled": true
}
}

But, kibana doesn't allow me to view the monitoring data of monitoring cluster/logstash, though it still allows viewing other cluster. We are not monitoring kibana though, but use it to view only the other clusters.

I'm able to see data of my 6.2 cluster, monitoring cluster and logstash using monitoring indices. But kibana is showing license issue for monitoring cluster and logstash, though the license sare valid.

Hope the above helps.

6

Thank you!! Testing now.

This is the config I have in my kibana.yml. It is the only thing I have changed from 6.2
xpack.security.enabled: false
xpack.graph.enabled: true
xpack.ml.enabled: true
xpack.monitoring.enabled: true
xpack.watcher.enabled: true
xpack.monitoring.kibana.collection.enabled: true
xpack.monitoring.ui.enabled: true

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

8

The error "Your Invalid license is inactive" (when the license is actually valid) can be caused by running a mismatched Kibana version.