Filter load monitoring

Elastic Stack Logstash
1

I have a lot of filters in logstash that deal with the processing of plain text logs and json.
With the linear growth of connected logs, the load increased and there was an increase in the queue.
In this connection, the question arose of monitoring and identifying the most costly filters for their further optimization.
However, I don't see how the API can help in this situation.
As I understand it, I need a request
curl -XGET 'localhost:9600/_node/stats/pipelines'

I get the following values

  "pipelines": {
    "main": {
      "events": {
        "in": 1105371893,
        "duration_in_millis": 1027801699,
        "out": 1105369863,
        "queue_push_duration_in_millis": 264106267,
        "filtered": 1105369863
      },
      "plugins": {
        "inputs": [
		...
        ],
		"filters": [
          {
            "id": "d777e68a451897b632b1544245f597a590ce21c2070dec49c93aaaf0911a92c3",
            "events": {
              "in": 7331741,
              "duration_in_millis": 2796,
              "out": 7331741
            },
            "name": "mutate"
          },
		  ...
          {
            "id": "ff24433329a5c1fdac94e79c47bfb197310a1e0d89963d97d787e6b9c4fc5c7c",
            "events": {
              "in": 0,
              "duration_in_millis": 267,
              "out": 0
            },
            "matches": 0,
            "failures": 0,
            "name": "grok",
            "patterns_per_field": {
              "[json][rest]": 1
            }
          },
....

How can I match these IDs with filters?
I need to understand which filters are the heaviest.

Filter file structure:
logstash/conf.d/
|_____ filter1.conf
|_____ filter2.conf
|_____ filterN.conf

filter structure:

filter{
    if "something" in [tags] {
        if "beats_input_codec_plain_applied" in  [tags] {
            grok {
        ....
}}}

Logstash 7.5.1

2

You can set the id in the filter configuration. It is one of the options that all filters support.

3

Thanks

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.