# Stack Monitoring fails to load logstash pipelines

**URL:** <https://discuss.elastic.co/t/stack-monitoring-fails-to-load-logstash-pipelines/281209>\
**Category:** Logstash\
**Tags:** elastic-stack-monitoring\
**Created:** [August 12, 2021, 1:48pm UTC](https://discuss.elastic.co/t/stack-monitoring-fails-to-load-logstash-pipelines/281209 "2021-08-12T13:48:06Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![radovan](https://avatars.discourse-cdn.com/v4/letter/r/2acd7d/32.png) [@radovan](https://discuss.elastic.co/u/radovan)\
**Post date:** [August 12, 2021, 1:48pm UTC](https://discuss.elastic.co/t/stack-monitoring-fails-to-load-logstash-pipelines/281209/1 "2021-08-12T13:48:06Z")

</div>

Hi,

I decided to create a monitoring cluster and after a few problems at the start all is running now. There are metricbeat clients on the logstash nodes with only logstash-xpack module enabled. Logs are flowing onto the monitoring cluster to .monitoring-logstash-\* index, but when I go to Stack Monitoring -\> Logstash - Pipelines (here I can see Events Emitted Rate for every pipeline) -\> and when I click on any pipeline, the wheel with "Loading" is just rolling and nothing happens.

Simple configuration of logstash node:

```auto
node.name: XXX
monitoring.cluster_uuid: XXX
monitoring.enabled: false
path.data: /var/lib/logstash
path.logs: /var/log/logstash
queue.max_bytes: 15gb
xpack.monitoring.enabled: false

```

Configuration of logstash-xpack module for metricbeat:

```auto
- module: logstash
  xpack.enabled: true
  period: 10s
  hosts: ["localhost:9600"]
  username: "metricbeat"
  password: XXX

```

User metricbeat used in module has roles remote\_monitoring\_collector and remote\_monitoring\_agent in the production cluster.

Output of metricbeat client is set to monitoring cluster's ingest/data node with username that has roles remote\_monitoring\_collector and remote\_monitoring\_agent in the monitoring cluster.

Monitoring of pipelines worked correctly while there was only legacy monitoring on the productiong cluster.

Version of monitoring and production cluster is 7.13.3

---

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [August 25, 2021, 2:01pm UTC](https://discuss.elastic.co/t/stack-monitoring-fails-to-load-logstash-pipelines/281209/2 "2021-08-25T14:01:05Z")

</div>

Hi @radovan,

If you open the browser dev tools, do you see any javascript errors?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [August 25, 2021, 2:13pm UTC](https://discuss.elastic.co/t/stack-monitoring-fails-to-load-logstash-pipelines/281209/3 "2021-08-25T14:13:25Z")

</div>

I have the same issue in my monitoring cluster with version 7.12.

Using inspect to see the javascript errors, this one keeps repeating:

```auto
TypeError: Cannot read property 'graph' of undefined
    at config_Config.update (monitoring.chunk.1.js:30)
    at pipeline_state_PipelineState.update (monitoring.chunk.1.js:30)
    at new pipeline_state_PipelineState (monitoring.chunk.1.js:30)
    at monitoring.chunk.1.js:30
    at c.$digest (kbn-ui-shared-deps.js:367)
    at c.$apply (kbn-ui-shared-deps.js:367)
    at monitoring.chunk.1.js:30
(anonymous) @ kbn-ui-shared-deps.js:367
(anonymous) @ kbn-ui-shared-deps.js:367
$digest @ kbn-ui-shared-deps.js:367
$apply @ kbn-ui-shared-deps.js:367
(anonymous) @ monitoring.chunk.1.js:30
Promise.then (async)
updateData @ monitoring.chunk.1.js:30
execute @ monitoring.chunk.1.js:30
(anonymous) @ monitoring.chunk.1.js:30
run @ monitoring.chunk.1.js:30
(anonymous) @ kbn-ui-shared-deps.js:367
Zi.completeTask @ kbn-ui-shared-deps.js:367
(anonymous) @ kbn-ui-shared-deps.js:367
setTimeout (async)
o.defer @ kbn-ui-shared-deps.js:367
a @ kbn-ui-shared-deps.js:367
start @ monitoring.chunk.1.js:30
reset @ monitoring.chunk.1.js:30
_ @ kbn-ui-shared-deps.js:367
(anonymous) @ kbn-ui-shared-deps.js:367
(anonymous) @ kbn-ui-shared-deps.js:367
(anonymous) @ kbn-ui-shared-deps.js:367
$digest @ kbn-ui-shared-deps.js:367
(anonymous) @ kbn-ui-shared-deps.js:367
Zi.completeTask @ kbn-ui-shared-deps.js:367
(anonymous) @ kbn-ui-shared-deps.js:367
setTimeout (async)
o.defer @ kbn-ui-shared-deps.js:367
$evalAsync @ kbn-ui-shared-deps.js:367
(anonymous) @ kbn-ui-shared-deps.js:367
u @ kbn-ui-shared-deps.js:367
then @ kbn-ui-shared-deps.js:367

```

---

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [August 25, 2021, 2:27pm UTC](https://discuss.elastic.co/t/stack-monitoring-fails-to-load-logstash-pipelines/281209/4 "2021-08-25T14:27:16Z")

</div>

Thanks @leandrojmp.

When you load the UI, are you able to [capture a HAR](https://support.zendesk.com/hc/en-us/articles/204410413-Generating-a-HAR-file-for-troubleshooting) when the error occurs?

---

<div class="post-metadata">

**Author:** ![radovan](https://avatars.discourse-cdn.com/v4/letter/r/2acd7d/32.png) [@radovan](https://discuss.elastic.co/u/radovan)\
**Post date:** [August 25, 2021, 2:47pm UTC](https://discuss.elastic.co/t/stack-monitoring-fails-to-load-logstash-pipelines/281209/5 "2021-08-25T14:47:21Z")

</div>

Hi @chrisronline,

yes, I see errors like these:

```auto
Content Security Policy: The page’s settings blocked the loading of a resource at inline (“script-src”). monitoring:342:1
TypeError: configJson is undefined
    update https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    update https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    pipeline_state_PipelineState https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    controller https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    $digest https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    $apply https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    updateData https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30

```

```auto
TypeError: configJson is undefined
    update https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    update https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    pipeline_state_PipelineState https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    controller https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    $digest https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    $apply https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    updateData https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    promise callback*base_controller_MonitoringViewBaseController/this.updateData https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    execute https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    run https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    run https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    c https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    completeTask https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    i https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
kbn-ui-shared-deps.js:375:78980
    a https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    get https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    $digest https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    $apply https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    updateData https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    (Async: promise callback)
    updateData https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    execute https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    run https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    run https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    c https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    completeTask https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    i https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375

```

And a lot of these:

```auto
TypeError: configJson is undefined
    update https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    update https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    pipeline_state_PipelineState https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    controller https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    $digest https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    $apply https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    updateData https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    promise callback*base_controller_MonitoringViewBaseController/this.updateData https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    execute https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    run https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    run https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    c https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    completeTask https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    i https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    setTimeout handler*cn/o.defer https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    s https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    start https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    reset https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    _ https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    finally https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    u https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    u https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    $digest https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    evalAsync https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    completeTask https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    i https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    setTimeout handler*cn/o.defer https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    $evalAsync https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    get https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    u https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    then https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    promise callback*run/< https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    run https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    c https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    completeTask https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    i https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375

```

---

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [August 25, 2021, 4:20pm UTC](https://discuss.elastic.co/t/stack-monitoring-fails-to-load-logstash-pipelines/281209/6 "2021-08-25T16:20:16Z")

</div>

Thanks @radovan

When you load the UI, are you able to [capture a HAR](https://support.zendesk.com/hc/en-us/articles/204410413-Generating-a-HAR-file-for-troubleshooting) when the error occurs?

---

<div class="post-metadata">

**Author:** ![radovan](https://avatars.discourse-cdn.com/v4/letter/r/2acd7d/32.png) [@radovan](https://discuss.elastic.co/u/radovan)\
**Post date:** [August 27, 2021, 10:20am UTC](https://discuss.elastic.co/t/stack-monitoring-fails-to-load-logstash-pipelines/281209/7 "2021-08-27T10:20:52Z")

</div>

Sure, here you go (pass is Har\_File):

[https://easyupload.io/gvdl8d](https://easyupload.io/gvdl8d)

---

<div class="post-metadata">

**Author:** ![radovan](https://avatars.discourse-cdn.com/v4/letter/r/2acd7d/32.png) [@radovan](https://discuss.elastic.co/u/radovan)\
**Post date:** [August 30, 2021, 10:34am UTC](https://discuss.elastic.co/t/stack-monitoring-fails-to-load-logstash-pipelines/281209/8 "2021-08-30T10:34:45Z")

</div>

Now after upgrading (metricbeat clients and monitoring cluster are on version 7.14.0) the pipeline monitoring of most recently added pipeline in logstash is working fine, but can't open any other pipeline monitoring because of the errors mentioned earlier.

---

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [August 30, 2021, 1:27pm UTC](https://discuss.elastic.co/t/stack-monitoring-fails-to-load-logstash-pipelines/281209/9 "2021-08-30T13:27:22Z")

</div>

Thanks for the HAR.

It looks like the pipeline isn't available in that time period, per the response:

```auto
 "payload": {
        "statusCode": 404,
        "error": "Not Found",
        "message": "Pipeline [joj-filebeat @] not found in the selected time range for cluster [WmRIg8hIR0yyXqLxAk7G1w]."
      },

```

Does that feel right? The page needs to handle that error better, but lets make sure that's the only bug at play here.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [August 30, 2021, 1:41pm UTC](https://discuss.elastic.co/t/stack-monitoring-fails-to-load-logstash-pipelines/281209/10 "2021-08-30T13:41:44Z")

</div>

Hello,

I've generated an `.har.` also, I can redact some information and share or I can look for some string in the file if you need.

Looking at your last response, I've searched for a similar error and found the same message:

```auto
Pipeline [fortigate-from-kafka @ ce763c1d0791a7139806b4a0c206a45807e8e9f56cbbab757ddfa2087b81b5df] not found in the selected time range for cluster [mrBKJ3WLQ6iLRBV9dxjm0Q]

```

This does not make sense, the pipeline has been running for days and time selected time range in the monitoring is the last 1 hour, also, this happens with every pipeline I have, none of them shows up.

---

<div class="post-metadata">

**Author:** ![radovan](https://avatars.discourse-cdn.com/v4/letter/r/2acd7d/32.png) [@radovan](https://discuss.elastic.co/u/radovan)\
**Post date:** [August 30, 2021, 1:59pm UTC](https://discuss.elastic.co/t/stack-monitoring-fails-to-load-logstash-pipelines/281209/11 "2021-08-30T13:59:35Z")

</div>

As @leandrojmp mentioned, all the pipelines are running for days or even months and only the last added pipeline is properly shown, all of the others got the error message in console.

---

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [August 31, 2021, 2:54pm UTC](https://discuss.elastic.co/t/stack-monitoring-fails-to-load-logstash-pipelines/281209/12 "2021-08-31T14:54:41Z")

</div>

Hmm okay. Let's run a query to see what the data looks like. There are several queries that run on the page load, but this single query should give us nearly all the data:

```auto
POST .monitoring-logstash-*/_search
{
  "size": 0,
  "aggs": {
    "types": {
      "terms": {
        "field": "type",
        "size": 100
      },
      "aggs": {
        "clusters": {
          "terms": {
            "field": "cluster_uuid",
            "size": 100
          },
          "aggs": {
            "pipelines_state": {
              "terms": {
                "field": "logstash_state.pipeline.id",
                "size": 100
              },
              "aggs": {
                "hashes": {
                  "terms": {
                    "field": "logstash_state.pipeline.hash",
                    "size": 100
                  },
                  "aggs": {
                    "latest": {
                      "max": {
                        "field": "timestamp"
                      }
                    },
                    "earliest": {
                      "min": {
                        "field": "timestamp"
                      }
                    }
                  }
                }
              }
            },
            "pipelines_stats": {
              "nested": {
                "path": "logstash_stats.pipelines"
              },
              "aggs": {
                "ids": {
                  "terms": {
                    "field": "logstash_stats.pipelines.id",
                    "size": 100
                  },
                  "aggs": {
                    "hashes": {
                      "terms": {
                        "field": "logstash_stats.pipelines.hash",
                        "size": 100
                      },
                      "aggs": {
                        "out": {
                          "reverse_nested": {},
                          "aggs": {
                            "latest": {
                              "max": {
                                "field": "timestamp"
                              }
                            },
                            "earliest": {
                              "min": {
                                "field": "timestamp"
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![radovan](https://avatars.discourse-cdn.com/v4/letter/r/2acd7d/32.png) [@radovan](https://discuss.elastic.co/u/radovan)\
**Post date:** [August 31, 2021, 3:10pm UTC](https://discuss.elastic.co/t/stack-monitoring-fails-to-load-logstash-pipelines/281209/13 "2021-08-31T15:10:43Z")

</div>

Here is the output (pass is 1fQuNJvi2K):

> **[Pastebin.com - Locked Paste](https://pastebin.com/R0mVmsUA)**
>
> Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

---

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [August 31, 2021, 4:50pm UTC](https://discuss.elastic.co/t/stack-monitoring-fails-to-load-logstash-pipelines/281209/14 "2021-08-31T16:50:44Z")

</div>

Thanks!

It seems there is an empty hash set for this pipeline (stored in `logstash_stats` type documents):

```auto
{
  "key" : "joj-filebeat",
  "doc_count" : 171595,
  "hashes" : {
    "doc_count_error_upper_bound" : 0,
    "sum_other_doc_count" : 0,
    "buckets" : [
      {
        "key" : "c721d5c548d724b22dce6d4903dc7b9edd837483136db544f591222dd473683b",
        "doc_count" : 171558,
        "out" : {
          "doc_count" : 171558,
          "earliest" : {
            "value" : 1.629849604225E12,
            "value_as_string" : "2021-08-25T00:00:04.225Z"
          },
          "latest" : {
            "value" : 1.630422362E12,
            "value_as_string" : "2021-08-31T15:06:02.000Z"
          }
        }
      },
      {
        "key" : "",
        "doc_count" : 37,
        "out" : {
          "doc_count" : 37,
          "earliest" : {
            "value" : 1.629884705464E12,
            "value_as_string" : "2021-08-25T09:45:05.464Z"
          },
          "latest" : {
            "value" : 1.630303700847E12,
            "value_as_string" : "2021-08-30T06:08:20.847Z"
          }
        }
      }
    ]
  }
},

```

I think what's happening is [this code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/monitoring/server/routes/api/v1/logstash/pipeline.js#L17) is selecting the empty hash, which doesn't have a corresponding entry in the other type of logstash monitoring documents `logstash_state`:

```auto
{
  "key" : "joj-filebeat",
  "doc_count" : 6,
  "hashes" : {
    "doc_count_error_upper_bound" : 0,
    "sum_other_doc_count" : 0,
    "buckets" : [
      {
        "key" : "c721d5c548d724b22dce6d4903dc7b9edd837483136db544f591222dd473683b",
        "doc_count" : 6,
        "earliest" : {
          "value" : 1.630319283755E12,
          "value_as_string" : "2021-08-30T10:28:03.755Z"
        },
        "latest" : {
          "value" : 1.630368006052E12,
          "value_as_string" : "2021-08-31T00:00:06.052Z"
        }
      }
    ]
  }
},

```

which is triggering [this code](https://github.com/elastic/kibana/blob/master/x-pack/plugins/monitoring/server/lib/logstash/get_pipeline.ts#L146) and returning the error.

One quick thing we can do to verify this is to try and call the API with the explicit hash:

`POST https://kibana-mon.XXX/api/monitoring/v1/clusters/WmRIg8hIR0yyXqLxAk7G1w/logstash/pipeline/joj-filebeat/c721d5c548d724b22dce6d4903dc7b9edd837483136db544f591222dd473683b`

Lemme know if that returns data and we'll go from there. Best to just paste the entire response.

Thanks

---

<div class="post-metadata">

**Author:** ![radovan](https://avatars.discourse-cdn.com/v4/letter/r/2acd7d/32.png) [@radovan](https://discuss.elastic.co/u/radovan)\
**Post date:** [August 31, 2021, 6:40pm UTC](https://discuss.elastic.co/t/stack-monitoring-fails-to-load-logstash-pipelines/281209/15 "2021-08-31T18:40:08Z")

</div>

How should I call the API? In browser it returns 404, in curl it says that "request body" is expected

---

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [August 31, 2021, 6:53pm UTC](https://discuss.elastic.co/t/stack-monitoring-fails-to-load-logstash-pipelines/281209/16 "2021-08-31T18:53:41Z")

</div>

Use Postman or cURL and send it as a `POST` with an empty JSON body `{}`

---

<div class="post-metadata">

**Author:** ![radovan](https://avatars.discourse-cdn.com/v4/letter/r/2acd7d/32.png) [@radovan](https://discuss.elastic.co/u/radovan)\
**Post date:** [September 3, 2021, 8:24am UTC](https://discuss.elastic.co/t/stack-monitoring-fails-to-load-logstash-pipelines/281209/17 "2021-09-03T08:24:00Z")

</div>

Here's the output (pass is VcNrk7GYQk):

> **[Pastebin.com - Locked Paste](https://pastebin.com/tLVPw2es)**
>
> Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

---

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [September 8, 2021, 5:47pm UTC](https://discuss.elastic.co/t/stack-monitoring-fails-to-load-logstash-pipelines/281209/18 "2021-09-08T17:47:09Z")

</div>

Yup, looks like this empty hash is the culprit. I don't know why this would happen though.

This might an issue for the Logstash team. One thing you can do is verify you see an empty hash in the Logstash monitoring APIs: [Node Stats API | Logstash Reference [7.14] | Elastic](https://www.elastic.co/guide/en/logstash/current/node-stats-api.html)

---

<div class="post-metadata">

**Author:** ![radovan](https://avatars.discourse-cdn.com/v4/letter/r/2acd7d/32.png) [@radovan](https://discuss.elastic.co/u/radovan)\
**Post date:** [September 8, 2021, 6:05pm UTC](https://discuss.elastic.co/t/stack-monitoring-fails-to-load-logstash-pipelines/281209/19 "2021-09-08T18:05:50Z")

</div>

Looking at the logs now, this appears on all the logstash nodes right after service startup:

```auto
[2021-09-08T13:10:05,564][ERROR][logstash.agent] Internal API server error {:status=>500, :request_method=>"GET", :path_info=>"/_node/stats/process", :query_string=>"pretty", :http_version=>"HTTP/1.1", :http_accept=>"*/*", :error=>"Unexpected Internal Error", :class=>"LogStash::Instrument::MetricStore::MetricNotFound", :message=>"For path: events. Map keys: [:reloads, :pipelines]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:241:in `block in get_recursively'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:240:in `get_recursively'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:251:in `block in get_recursively'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:240:in `get_recursively'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:111:in `block in get'", "org/jruby/ext/thread/Mutex.java:164:in `synchronize'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:110:in `get'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:124:in `get_shallow'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:173:in `block in extract_metrics'", "org/jruby/RubyArray.java:1809:in `each'", "org/jruby/RubyEnumerable.java:1126:in `inject'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:149:in `extract_metrics'", "/usr/share/logstash/logstash-core/lib/logstash/api/service.rb:45:in `extract_metrics'", "/usr/share/logstash/logstash-core/lib/logstash/api/commands/base.rb:37:in `extract_metrics'", "/usr/share/logstash/logstash-core/lib/logstash/api/commands/stats.rb:73:in `events'", "/usr/share/logstash/logstash-core/lib/logstash/api/modules/node_stats.rb:57:in `events_payload'", "/usr/share/logstash/logstash-core/lib/logstash/api/modules/node_stats.rb:37:in `block in GET /?:filter?'", "org/jruby/RubyMethod.java:115:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `block in compile!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block in route!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1032:in `route_eval'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block in route!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1061:in `block in process_route'", "org/jruby/RubyKernel.java:1189:in `catch'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `process_route'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1011:in `block in route!'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `route!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1129:in `block in dispatch!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'", "org/jruby/RubyKernel.java:1189:in `catch'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1124:in `dispatch!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `block in call!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'", "org/jruby/RubyKernel.java:1189:in `catch'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `call!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:929:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-protection-2.1.0/lib/rack/protection/frame_options.rb:31:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'", "/usr/share/logstash/logstash-core/lib/logstash/api/rack_app.rb:74:in `call'", "/usr/share/logstash/logstash-core/lib/logstash/api/rack_app.rb:48:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-2.2.3/lib/rack/builder.rb:244:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/puma-4.3.8-java/lib/puma/server.rb:718:in `handle_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/puma-4.3.8-java/lib/puma/server.rb:472:in `process_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/puma-4.3.8-java/lib/puma/server.rb:328:in `block in run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/puma-4.3.8-java/lib/puma/thread_pool.rb:134:in `block in spawn_thread'"]}

```

```auto
[2021-09-08T13:10:18,528][ERROR][logstash.agent] Internal API server error {:status=>500, :request_method=>"GET", :path_info=>"/_node/stats", :query_string=>"vertices=true", :http_version=>"HTTP/1.1", :http_accept=>nil, :error=>"Unexpected Internal Error", :class=>"LogStash::Instrument::MetricStore::MetricNotFound", :message=>"For path: events. Map keys: [:reloads, :pipelines]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:241:in `block in get_recursively'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:240:in `get_recursively'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:251:in `block in get_recursively'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:240:in `get_recursively'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:111:in `block in get'", "org/jruby/ext/thread/Mutex.java:164:in `synchronize'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:110:in `get'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:124:in `get_shallow'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:173:in `block in extract_metrics'", "org/jruby/RubyArray.java:1809:in `each'", "org/jruby/RubyEnumerable.java:1126:in `inject'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:149:in `extract_metrics'", "/usr/share/logstash/logstash-core/lib/logstash/api/service.rb:45:in `extract_metrics'", "/usr/share/logstash/logstash-core/lib/logstash/api/commands/base.rb:37:in `extract_metrics'", "/usr/share/logstash/logstash-core/lib/logstash/api/commands/stats.rb:73:in `events'", "/usr/share/logstash/logstash-core/lib/logstash/api/modules/node_stats.rb:57:in `events_payload'", "/usr/share/logstash/logstash-core/lib/logstash/api/modules/node_stats.rb:37:in `block in NodeStats'", "org/jruby/RubyMethod.java:115:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `block in compile!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block in route!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1032:in `route_eval'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block in route!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1061:in `block in process_route'", "org/jruby/RubyKernel.java:1189:in `catch'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `process_route'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1011:in `block in route!'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `route!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1129:in `block in dispatch!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'", "org/jruby/RubyKernel.java:1189:in `catch'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1124:in `dispatch!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `block in call!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'", "org/jruby/RubyKernel.java:1189:in `catch'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `call!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:929:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-protection-2.1.0/lib/rack/protection/frame_options.rb:31:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'", "/usr/share/logstash/logstash-core/lib/logstash/api/rack_app.rb:74:in `call'", "/usr/share/logstash/logstash-core/lib/logstash/api/rack_app.rb:48:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-2.2.3/lib/rack/builder.rb:244:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/puma-4.3.8-java/lib/puma/server.rb:718:in `handle_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/puma-4.3.8-java/lib/puma/server.rb:472:in `process_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/puma-4.3.8-java/lib/puma/server.rb:328:in `block in run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/puma-4.3.8-java/lib/puma/thread_pool.rb:134:in `block in spawn_thread'"]}

```

What could be causing this?

---

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [September 8, 2021, 6:17pm UTC](https://discuss.elastic.co/t/stack-monitoring-fails-to-load-logstash-pipelines/281209/20 "2021-09-08T18:17:30Z")

</div>

I'm not sure unfortunately - I don't work on the Logstash product. I've transferred this discuss post there and someone on their team should pick it up soon and be able to help.

[Next page](https://discuss.elastic.co/t/stack-monitoring-fails-to-load-logstash-pipelines/281209.md?page=2)