# Filter load monitoring

**URL:** <https://discuss.elastic.co/t/filter-load-monitoring/300461>\
**Category:** Logstash\
**Created:** [March 23, 2022, 1:43pm UTC](https://discuss.elastic.co/t/filter-load-monitoring/300461 "2022-03-23T13:43:34Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dargod](https://avatars.discourse-cdn.com/v4/letter/d/e36b37/32.png) [@Dargod](https://discuss.elastic.co/u/Dargod)\
**Post date:** [March 23, 2022, 1:43pm UTC](https://discuss.elastic.co/t/filter-load-monitoring/300461/1 "2022-03-23T13:43:34Z")

</div>

I have a lot of filters in logstash that deal with the processing of plain text logs and json.  
With the linear growth of connected logs, the load increased and there was an increase in the queue.  
In this connection, the question arose of monitoring and identifying the most costly filters for their further optimization.  
However, I don't see how the API can help in this situation.  
As I understand it, I need a request  
curl -XGET 'localhost:9600/\_node/stats/pipelines'

I get the following values

```auto
  "pipelines": {
    "main": {
      "events": {
        "in": 1105371893,
        "duration_in_millis": 1027801699,
        "out": 1105369863,
        "queue_push_duration_in_millis": 264106267,
        "filtered": 1105369863
      },
      "plugins": {
        "inputs": [
		...
        ],
		"filters": [
          {
            "id": "d777e68a451897b632b1544245f597a590ce21c2070dec49c93aaaf0911a92c3",
            "events": {
              "in": 7331741,
              "duration_in_millis": 2796,
              "out": 7331741
            },
            "name": "mutate"
          },
		  ...
          {
            "id": "ff24433329a5c1fdac94e79c47bfb197310a1e0d89963d97d787e6b9c4fc5c7c",
            "events": {
              "in": 0,
              "duration_in_millis": 267,
              "out": 0
            },
            "matches": 0,
            "failures": 0,
            "name": "grok",
            "patterns_per_field": {
              "[json][rest]": 1
            }
          },
....

```

How can I match these IDs with filters?  
I need to understand which filters are the heaviest.

Filter file structure:  
logstash/conf.d/  
|\_\_\_\_\_ filter1.conf  
|\_\_\_\_\_ filter2.conf  
|\_\_\_\_\_ filterN.conf

filter structure:

```auto
filter{
    if "something" in [tags] {
        if "beats_input_codec_plain_applied" in [tags] {
            grok {
        ....
}}}

```

Logstash 7.5.1

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 23, 2022, 3:58pm UTC](https://discuss.elastic.co/t/filter-load-monitoring/300461/2 "2022-03-23T15:58:52Z")

</div>

You can set the id in the [filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-id) configuration. It is one of the options that all filters support.

---

<div class="post-metadata">

**Author:** ![Dargod](https://avatars.discourse-cdn.com/v4/letter/d/e36b37/32.png) [@Dargod](https://discuss.elastic.co/u/Dargod)\
**Post date:** [March 24, 2022, 5:55pm UTC](https://discuss.elastic.co/t/filter-load-monitoring/300461/3 "2022-03-24T17:55:53Z")

</div>

Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 21, 2022, 5:56pm UTC](https://discuss.elastic.co/t/filter-load-monitoring/300461/4 "2022-04-21T17:56:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.