Stack Monitoring fails to load logstash pipelines

Elastic Stack Logstash
1

Hi,

I decided to create a monitoring cluster and after a few problems at the start all is running now. There are metricbeat clients on the logstash nodes with only logstash-xpack module enabled. Logs are flowing onto the monitoring cluster to .monitoring-logstash-* index, but when I go to Stack Monitoring -> Logstash - Pipelines (here I can see Events Emitted Rate for every pipeline) -> and when I click on any pipeline, the wheel with "Loading" is just rolling and nothing happens.

Simple configuration of logstash node:

node.name: XXX
monitoring.cluster_uuid: XXX
monitoring.enabled: false
path.data: /var/lib/logstash
path.logs: /var/log/logstash
queue.max_bytes: 15gb
xpack.monitoring.enabled: false

Configuration of logstash-xpack module for metricbeat:

- module: logstash
  xpack.enabled: true
  period: 10s
  hosts: ["localhost:9600"]
  username: "metricbeat"
  password: XXX

User metricbeat used in module has roles remote_monitoring_collector and remote_monitoring_agent in the production cluster.

Output of metricbeat client is set to monitoring cluster's ingest/data node with username that has roles remote_monitoring_collector and remote_monitoring_agent in the monitoring cluster.

Monitoring of pipelines worked correctly while there was only legacy monitoring on the productiong cluster.

Version of monitoring and production cluster is 7.13.3

2

Hi @radovan,

If you open the browser dev tools, do you see any javascript errors?

3

I have the same issue in my monitoring cluster with version 7.12.

Using inspect to see the javascript errors, this one keeps repeating:

TypeError: Cannot read property 'graph' of undefined
    at config_Config.update (monitoring.chunk.1.js:30)
    at pipeline_state_PipelineState.update (monitoring.chunk.1.js:30)
    at new pipeline_state_PipelineState (monitoring.chunk.1.js:30)
    at monitoring.chunk.1.js:30
    at c.$digest (kbn-ui-shared-deps.js:367)
    at c.$apply (kbn-ui-shared-deps.js:367)
    at monitoring.chunk.1.js:30
(anonymous) @ kbn-ui-shared-deps.js:367
(anonymous) @ kbn-ui-shared-deps.js:367
$digest @ kbn-ui-shared-deps.js:367
$apply @ kbn-ui-shared-deps.js:367
(anonymous) @ monitoring.chunk.1.js:30
Promise.then (async)
updateData @ monitoring.chunk.1.js:30
execute @ monitoring.chunk.1.js:30
(anonymous) @ monitoring.chunk.1.js:30
run @ monitoring.chunk.1.js:30
(anonymous) @ kbn-ui-shared-deps.js:367
Zi.completeTask @ kbn-ui-shared-deps.js:367
(anonymous) @ kbn-ui-shared-deps.js:367
setTimeout (async)
o.defer @ kbn-ui-shared-deps.js:367
a @ kbn-ui-shared-deps.js:367
start @ monitoring.chunk.1.js:30
reset @ monitoring.chunk.1.js:30
_ @ kbn-ui-shared-deps.js:367
(anonymous) @ kbn-ui-shared-deps.js:367
(anonymous) @ kbn-ui-shared-deps.js:367
(anonymous) @ kbn-ui-shared-deps.js:367
$digest @ kbn-ui-shared-deps.js:367
(anonymous) @ kbn-ui-shared-deps.js:367
Zi.completeTask @ kbn-ui-shared-deps.js:367
(anonymous) @ kbn-ui-shared-deps.js:367
setTimeout (async)
o.defer @ kbn-ui-shared-deps.js:367
$evalAsync @ kbn-ui-shared-deps.js:367
(anonymous) @ kbn-ui-shared-deps.js:367
u @ kbn-ui-shared-deps.js:367
then @ kbn-ui-shared-deps.js:367
4

Thanks @leandrojmp.

When you load the UI, are you able to capture a HAR when the error occurs?

5

Hi @chrisronline,

yes, I see errors like these:

Content Security Policy: The page’s settings blocked the loading of a resource at inline (“script-src”). monitoring:342:1
TypeError: configJson is undefined
    update https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    update https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    pipeline_state_PipelineState https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    controller https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    $digest https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    $apply https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    updateData https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30

TypeError: configJson is undefined
    update https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    update https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    pipeline_state_PipelineState https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    controller https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    $digest https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    $apply https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    updateData https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    promise callback*base_controller_MonitoringViewBaseController/this.updateData https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    execute https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    run https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    run https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    c https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    completeTask https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    i https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
kbn-ui-shared-deps.js:375:78980
    a https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    get https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    $digest https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    $apply https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    updateData https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    (Async: promise callback)
    updateData https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    execute https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    run https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    run https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    c https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    completeTask https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    i https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375

And a lot of these:

TypeError: configJson is undefined
    update https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    update https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    pipeline_state_PipelineState https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    controller https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    $digest https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    $apply https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    updateData https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    promise callback*base_controller_MonitoringViewBaseController/this.updateData https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    execute https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    run https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    run https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    c https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    completeTask https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    i https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    setTimeout handler*cn/o.defer https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    s https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    start https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    reset https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    _ https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    finally https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    u https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    u https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    $digest https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    evalAsync https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    completeTask https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    i https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    setTimeout handler*cn/o.defer https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    $evalAsync https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    get https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    u https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    then https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    promise callback*run/< https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    run https://kibana-mon.XXX/41006/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30
    c https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    completeTask https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
    i https://kibana-mon.XXX/41006/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375
6

Thanks @radovan

When you load the UI, are you able to capture a HAR when the error occurs?

7

Sure, here you go (pass is Har_File):

https://easyupload.io/gvdl8d

8

Now after upgrading (metricbeat clients and monitoring cluster are on version 7.14.0) the pipeline monitoring of most recently added pipeline in logstash is working fine, but can't open any other pipeline monitoring because of the errors mentioned earlier.

9

Thanks for the HAR.

It looks like the pipeline isn't available in that time period, per the response:

 "payload": {
        "statusCode": 404,
        "error": "Not Found",
        "message": "Pipeline [joj-filebeat @ ] not found in the selected time range for cluster [WmRIg8hIR0yyXqLxAk7G1w]."
      },

Does that feel right? The page needs to handle that error better, but lets make sure that's the only bug at play here.

10

Hello,

I've generated an .har. also, I can redact some information and share or I can look for some string in the file if you need.

Looking at your last response, I've searched for a similar error and found the same message:

Pipeline [fortigate-from-kafka @ ce763c1d0791a7139806b4a0c206a45807e8e9f56cbbab757ddfa2087b81b5df] not found in the selected time range for cluster [mrBKJ3WLQ6iLRBV9dxjm0Q]

This does not make sense, the pipeline has been running for days and time selected time range in the monitoring is the last 1 hour, also, this happens with every pipeline I have, none of them shows up.

11

As @leandrojmp mentioned, all the pipelines are running for days or even months and only the last added pipeline is properly shown, all of the others got the error message in console.

12

Hmm okay. Let's run a query to see what the data looks like. There are several queries that run on the page load, but this single query should give us nearly all the data:

POST .monitoring-logstash-*/_search
{
  "size": 0,
  "aggs": {
    "types": {
      "terms": {
        "field": "type",
        "size": 100
      },
      "aggs": {
        "clusters": {
          "terms": {
            "field": "cluster_uuid",
            "size": 100
          },
          "aggs": {
            "pipelines_state": {
              "terms": {
                "field": "logstash_state.pipeline.id",
                "size": 100
              },
              "aggs": {
                "hashes": {
                  "terms": {
                    "field": "logstash_state.pipeline.hash",
                    "size": 100
                  },
                  "aggs": {
                    "latest": {
                      "max": {
                        "field": "timestamp"
                      }
                    },
                    "earliest": {
                      "min": {
                        "field": "timestamp"
                      }
                    }
                  }
                }
              }
            },
            "pipelines_stats": {
              "nested": {
                "path": "logstash_stats.pipelines"
              },
              "aggs": {
                "ids": {
                  "terms": {
                    "field": "logstash_stats.pipelines.id",
                    "size": 100
                  },
                  "aggs": {
                    "hashes": {
                      "terms": {
                        "field": "logstash_stats.pipelines.hash",
                        "size": 100
                      },
                      "aggs": {
                        "out": {
                          "reverse_nested": {},
                          "aggs": {
                            "latest": {
                              "max": {
                                "field": "timestamp"
                              }
                            },
                            "earliest": {
                              "min": {
                                "field": "timestamp"
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}
13

Here is the output (pass is 1fQuNJvi2K):

14

Thanks!

It seems there is an empty hash set for this pipeline (stored in logstash_stats type documents):

{
  "key" : "joj-filebeat",
  "doc_count" : 171595,
  "hashes" : {
    "doc_count_error_upper_bound" : 0,
    "sum_other_doc_count" : 0,
    "buckets" : [
      {
        "key" : "c721d5c548d724b22dce6d4903dc7b9edd837483136db544f591222dd473683b",
        "doc_count" : 171558,
        "out" : {
          "doc_count" : 171558,
          "earliest" : {
            "value" : 1.629849604225E12,
            "value_as_string" : "2021-08-25T00:00:04.225Z"
          },
          "latest" : {
            "value" : 1.630422362E12,
            "value_as_string" : "2021-08-31T15:06:02.000Z"
          }
        }
      },
      {
        "key" : "",
        "doc_count" : 37,
        "out" : {
          "doc_count" : 37,
          "earliest" : {
            "value" : 1.629884705464E12,
            "value_as_string" : "2021-08-25T09:45:05.464Z"
          },
          "latest" : {
            "value" : 1.630303700847E12,
            "value_as_string" : "2021-08-30T06:08:20.847Z"
          }
        }
      }
    ]
  }
},

I think what's happening is this code is selecting the empty hash, which doesn't have a corresponding entry in the other type of logstash monitoring documents logstash_state:

{
  "key" : "joj-filebeat",
  "doc_count" : 6,
  "hashes" : {
    "doc_count_error_upper_bound" : 0,
    "sum_other_doc_count" : 0,
    "buckets" : [
      {
        "key" : "c721d5c548d724b22dce6d4903dc7b9edd837483136db544f591222dd473683b",
        "doc_count" : 6,
        "earliest" : {
          "value" : 1.630319283755E12,
          "value_as_string" : "2021-08-30T10:28:03.755Z"
        },
        "latest" : {
          "value" : 1.630368006052E12,
          "value_as_string" : "2021-08-31T00:00:06.052Z"
        }
      }
    ]
  }
},

which is triggering this code and returning the error.

One quick thing we can do to verify this is to try and call the API with the explicit hash:

POST https://kibana-mon.XXX/api/monitoring/v1/clusters/WmRIg8hIR0yyXqLxAk7G1w/logstash/pipeline/joj-filebeat/c721d5c548d724b22dce6d4903dc7b9edd837483136db544f591222dd473683b

Lemme know if that returns data and we'll go from there. Best to just paste the entire response.

Thanks

15

How should I call the API? In browser it returns 404, in curl it says that "request body" is expected

16

Use Postman or cURL and send it as a POST with an empty JSON body {}

17

Here's the output (pass is VcNrk7GYQk):

18

Yup, looks like this empty hash is the culprit. I don't know why this would happen though.

This might an issue for the Logstash team. One thing you can do is verify you see an empty hash in the Logstash monitoring APIs: Node Stats API | Logstash Reference [7.14] | Elastic

19

Looking at the logs now, this appears on all the logstash nodes right after service startup:

[2021-09-08T13:10:05,564][ERROR][logstash.agent           ] Internal API server error {:status=>500, :request_method=>"GET", :path_info=>"/_node/stats/process", :query_string=>"pretty", :http_version=>"HTTP/1.1", :http_accept=>"*/*", :error=>"Unexpected Internal Error", :class=>"LogStash::Instrument::MetricStore::MetricNotFound", :message=>"For path: events. Map keys: [:reloads, :pipelines]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:241:in `block in get_recursively'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:240:in `get_recursively'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:251:in `block in get_recursively'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:240:in `get_recursively'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:111:in `block in get'", "org/jruby/ext/thread/Mutex.java:164:in `synchronize'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:110:in `get'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:124:in `get_shallow'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:173:in `block in extract_metrics'", "org/jruby/RubyArray.java:1809:in `each'", "org/jruby/RubyEnumerable.java:1126:in `inject'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:149:in `extract_metrics'", "/usr/share/logstash/logstash-core/lib/logstash/api/service.rb:45:in `extract_metrics'", "/usr/share/logstash/logstash-core/lib/logstash/api/commands/base.rb:37:in `extract_metrics'", "/usr/share/logstash/logstash-core/lib/logstash/api/commands/stats.rb:73:in `events'", "/usr/share/logstash/logstash-core/lib/logstash/api/modules/node_stats.rb:57:in `events_payload'", "/usr/share/logstash/logstash-core/lib/logstash/api/modules/node_stats.rb:37:in `block in GET /?:filter?'", "org/jruby/RubyMethod.java:115:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `block in compile!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block in route!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1032:in `route_eval'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block in route!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1061:in `block in process_route'", "org/jruby/RubyKernel.java:1189:in `catch'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `process_route'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1011:in `block in route!'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `route!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1129:in `block in dispatch!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'", "org/jruby/RubyKernel.java:1189:in `catch'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1124:in `dispatch!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `block in call!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'", "org/jruby/RubyKernel.java:1189:in `catch'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `call!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:929:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-protection-2.1.0/lib/rack/protection/frame_options.rb:31:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'", "/usr/share/logstash/logstash-core/lib/logstash/api/rack_app.rb:74:in `call'", "/usr/share/logstash/logstash-core/lib/logstash/api/rack_app.rb:48:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-2.2.3/lib/rack/builder.rb:244:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/puma-4.3.8-java/lib/puma/server.rb:718:in `handle_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/puma-4.3.8-java/lib/puma/server.rb:472:in `process_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/puma-4.3.8-java/lib/puma/server.rb:328:in `block in run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/puma-4.3.8-java/lib/puma/thread_pool.rb:134:in `block in spawn_thread'"]}

[2021-09-08T13:10:18,528][ERROR][logstash.agent           ] Internal API server error {:status=>500, :request_method=>"GET", :path_info=>"/_node/stats", :query_string=>"vertices=true", :http_version=>"HTTP/1.1", :http_accept=>nil, :error=>"Unexpected Internal Error", :class=>"LogStash::Instrument::MetricStore::MetricNotFound", :message=>"For path: events. Map keys: [:reloads, :pipelines]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:241:in `block in get_recursively'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:240:in `get_recursively'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:251:in `block in get_recursively'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:240:in `get_recursively'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:111:in `block in get'", "org/jruby/ext/thread/Mutex.java:164:in `synchronize'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:110:in `get'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:124:in `get_shallow'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:173:in `block in extract_metrics'", "org/jruby/RubyArray.java:1809:in `each'", "org/jruby/RubyEnumerable.java:1126:in `inject'", "/usr/share/logstash/logstash-core/lib/logstash/instrument/metric_store.rb:149:in `extract_metrics'", "/usr/share/logstash/logstash-core/lib/logstash/api/service.rb:45:in `extract_metrics'", "/usr/share/logstash/logstash-core/lib/logstash/api/commands/base.rb:37:in `extract_metrics'", "/usr/share/logstash/logstash-core/lib/logstash/api/commands/stats.rb:73:in `events'", "/usr/share/logstash/logstash-core/lib/logstash/api/modules/node_stats.rb:57:in `events_payload'", "/usr/share/logstash/logstash-core/lib/logstash/api/modules/node_stats.rb:37:in `block in NodeStats'", "org/jruby/RubyMethod.java:115:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `block in compile!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block in route!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1032:in `route_eval'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block in route!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1061:in `block in process_route'", "org/jruby/RubyKernel.java:1189:in `catch'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `process_route'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1011:in `block in route!'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `route!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1129:in `block in dispatch!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'", "org/jruby/RubyKernel.java:1189:in `catch'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1124:in `dispatch!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `block in call!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'", "org/jruby/RubyKernel.java:1189:in `catch'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `call!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:929:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-protection-2.1.0/lib/rack/protection/frame_options.rb:31:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'", "org/jruby/RubyArray.java:1809:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'", "/usr/share/logstash/logstash-core/lib/logstash/api/rack_app.rb:74:in `call'", "/usr/share/logstash/logstash-core/lib/logstash/api/rack_app.rb:48:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/rack-2.2.3/lib/rack/builder.rb:244:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/puma-4.3.8-java/lib/puma/server.rb:718:in `handle_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/puma-4.3.8-java/lib/puma/server.rb:472:in `process_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/puma-4.3.8-java/lib/puma/server.rb:328:in `block in run'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/puma-4.3.8-java/lib/puma/thread_pool.rb:134:in `block in spawn_thread'"]}

What could be causing this?

20

I'm not sure unfortunately - I don't work on the Logstash product. I've transferred this discuss post there and someone on their team should pick it up soon and be able to help.