Stack Monitoring: Missing Logstash Pipeline Stats

[tomx1]

We have a 10 Node Elasticsearch cluster and one single node "cluster" to store all metrics. Those metrics are collected by metricbeat. We ship data into this cluster mainly with logstash. Logstash stats are also gathered by an dedicated metricbeat instance. To view all those metrics we use the Stack Monitoring App in Kibana. This works fine, except for the pipeline stats of logstash. Unfortunately only an empty page is shown if clicking on one of the active pipelines:

Security is enabled on both Clusters. If i directly search on the monitoring node via curl then i see all the pipeline stats data
curl -k -u user:password https://localhost:9200/.monitoring-logstash-7-mb-2020.03.10/_search?pretty

Any Idea what the problem could be?

[rashmi (kulkarni)]

On what version of the stack are you on? There were some known issues on 7.3.0 :Please refer here if you are on that version: https://github.com/elastic/support-known-issues/issues/1

cc @chrisronline

Thanks
Rashmi

[tomx1]

Hi rashmi... we are currently on 7.6.0

[Igor_Zaytsev]

@tomx1 I don’t think this is a security issue, since everything else seems to be working fine. We do however have an issue where if you explicitly set enable_metric to false the pipeline page will be empty: https://github.com/elastic/kibana/issues/44626 which happens in version 6.8 and above.

Are you seeing any errors in the browser's console? Or, any warnings/errors in ES, LS, and KB consoles?

[tomx1]

I don't use the enable_metric option but yes, I see the following errors in the browser console (chrome):

Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.

bootstrap.js:9 ^ A single error about an inline script not firing due to content security policy is expected!
vendors_1.bundle.dll.js:113 INFO: 2020-03-18T15:15:53Z
  Adding connection to https://example.com/elasticsearch


kbn-ui-shared-deps.js:368 TypeError: Cannot read property 'graph' of undefined
    at Config.update (monitoring.bundle.js:3)
    at PipelineState.update (monitoring.bundle.js:3)
    at new PipelineState (monitoring.bundle.js:3)
    at monitoring.bundle.js:3
    at u.$digest (kbn-ui-shared-deps.js:368)
    at u.$apply (kbn-ui-shared-deps.js:368)
    at monitoring.bundle.js:3

[Igor_Zaytsev]

This might be a result of a known issue with long running queries noted here: https://github.com/elastic/kibana/issues/57327

Can you please try one of the following:

• Increase (or decrease) the time range at the top right
• Increasing elasticsearch.requestTimeout setting in your kibana.yml file

Let me know if any of these work

[tomx1]

Unfortunately no success. I've increased request timeout to 60000 and tried several time ranges.

[chrisronline (Chris Roberson)]

Hi @tomx1,

Can you run this query against the monitoring cluster and return the results?

POST .monitoring-logstash-*/_search?filter_path=hits.hits._source.logstash_state
{
  "size": 1,
  "sort": {
    "timestamp": {
      "order": "desc"
    }
  },
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "type": "logstash_state"
          }
        }
      ]
    }
  }
}