# Grok (or any alternative) to search for keywords in logs

**URL:** <https://discuss.elastic.co/t/grok-or-any-alternative-to-search-for-keywords-in-logs/327351>\
**Category:** Logstash\
**Created:** [March 9, 2023, 11:50am UTC](https://discuss.elastic.co/t/grok-or-any-alternative-to-search-for-keywords-in-logs/327351 "2023-03-09T11:50:30Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mark\_S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_s/32/98686_2.png) [@Mark\_S](https://discuss.elastic.co/u/Mark_S)\
**Post date:** [March 9, 2023, 11:50am UTC](https://discuss.elastic.co/t/grok-or-any-alternative-to-search-for-keywords-in-logs/327351/1 "2023-03-09T11:50:31Z")

</div>

Hello,

Is it possible to create keywords in logstash, by searching for them in the message?

The logs are formatted in the following way, however they are not always in the same place - they could be embedded in other messages. For example in the following I would like to create keyword-value pairs  
protocol=HTTP/1.1  
uname=test-camel-svc  
method=PUT  
registert=02384827  
etc..

```auto
INFO Test-Info:[protocol=HTTP/1.1, uname=test-camel-svc, method=PUT, registert=02384827, server=TESTVM9:8083, tracker_id=84d9d0231-2231-4a11-5e2d-88afa5ee12c6, bda_id=4352, testbda_id=?/?, cause=incoming_request, calling_server=XX.XXX.XXX.XXX, request_time=N/A, reception_date=2020-01-00T01:00:00.000+0000, time_elapsed_ms=37] 3123344 --- [XNIO-1 task-1] g.u.m.commons.logging.MDCLoggingFilter : processing_end, processing_end

```

Note that the logs could be something completely irrelevant (for example a java error) or ( **and this is the problem** ): the whole info could be included in other messages with slightly different format:

```auto
WARN -- extra-characters .blahblah Test-Info:[protocol=HTTP/1.1, uname=test-camel-svc, method=PUT, registert=02384827, server=TESTVM9:8083, tracker_id=84d9d0231-2231-4a11-5e2d-88afa5ee12c6, bda_id=4352, testbda_id=?/?, cause=incoming_request, calling_server=XX.XXX.XXX.XXX, request_time=N/A, reception_date=2020-01-00T01:00:00.000+0000, time_elapsed_ms=37] 3123344 --- [XNIO-1 task-1] g.u.m.commons.logging.MDCLoggingFilter : processing_end, processing_end some more-characters-here

```

Is there any way to do this with grok (or any alternative to grok?)

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [March 9, 2023, 2:51pm UTC](https://discuss.elastic.co/t/grok-or-any-alternative-to-search-for-keywords-in-logs/327351/2 "2023-03-09T14:51:12Z")

</div>

You want what is inside the square brackets after `Test-Info:` right?

For example, in this message:

```auto
INFO Test-Info:[protocol=HTTP/1.1, uname=test-camel-svc, method=PUT, registert=02384827, server=TESTVM9:8083, tracker_id=84d9d0231-2231-4a11-5e2d-88afa5ee12c6, bda_id=4352, testbda_id=?/?, cause=incoming_request, calling_server=XX.XXX.XXX.XXX, request_time=N/A, reception_date=2020-01-00T01:00:00.000+0000, time_elapsed_ms=37] 3123344 --- [XNIO-1 task-1] g.u.m.commons.logging.MDCLoggingFilter : processing_end, processing_end

```

You want these fields:

```auto
protocol=HTTP/1.1, uname=test-camel-svc, method=PUT, registert=02384827, server=TESTVM9:8083, tracker_id=84d9d0231-2231-4a11-5e2d-88afa5ee12c6, bda_id=4352, testbda_id=?/?, cause=incoming_request, calling_server=XX.XXX.XXX.XXX, request_time=N/A, reception_date=2020-01-00T01:00:00.000+0000, time_elapsed_ms=37

```

If so, you can use a combination of `dissect` and `kv`.

```auto
filter {
    dissect {
        mapping => {
            "message" => "%{}Test-Info:[%{kvMsg}]%{}"
        }
    }
    kv {
        source => "kvMsg"
        value_split => "="
        field_split => ", "
    }
}

```

The `dissect` filter above you put everything between the square brackets after `Test-Info` in a field called _kvMsg_, the `kv` filter will then parse this message and create the individual fields.

---

<div class="post-metadata">

**Author:** ![Mark\_S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_s/32/98686_2.png) [@Mark\_S](https://discuss.elastic.co/u/Mark_S)\
**Post date:** [March 9, 2023, 4:30pm UTC](https://discuss.elastic.co/t/grok-or-any-alternative-to-search-for-keywords-in-logs/327351/3 "2023-03-09T16:30:39Z")

</div>

interesting! The problem is that one extra space can break this ... It is possible to have an extra space or slight variations somehow?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [March 9, 2023, 7:19pm UTC](https://discuss.elastic.co/t/grok-or-any-alternative-to-search-for-keywords-in-logs/327351/4 "2023-03-09T19:19:27Z")

</div>

> [@Mark\_S](#):
>
> The problem is that one extra space can break this ... It is possible to have an extra space or slight variations somehow?

Extra space where? Can you share an example?

---

<div class="post-metadata">

**Author:** ![Sunile\_Manjee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunile_manjee/32/111461_2.png) [@Sunile\_Manjee](https://discuss.elastic.co/u/Sunile_Manjee)\
**Post date:** [March 11, 2023, 4:40am UTC](https://discuss.elastic.co/t/grok-or-any-alternative-to-search-for-keywords-in-logs/327351/5 "2023-03-11T04:40:53Z")

</div>

> [@Mark\_S](#):
>
> r

I haven't fully tested this but curious if it will work for you. This assumes `Test-Info` is in your log message.

```auto
filter {
  if "Test-Info" in [message] {
    kv {
      source => "message"
      field_split => ", "
      value_split => "="
      trim_key => " "
      trim_value => " "
      prefix => ""
    }
  }
}

```

it should ignore log messages that do not match your desired pattern and handle extra spaces.

---

<div class="post-metadata">

**Author:** ![Mark\_S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_s/32/98686_2.png) [@Mark\_S](https://discuss.elastic.co/u/Mark_S)\
**Post date:** [March 11, 2023, 7:31am UTC](https://discuss.elastic.co/t/grok-or-any-alternative-to-search-for-keywords-in-logs/327351/6 "2023-03-11T07:31:48Z")

</div>

> [@Sunile\_Manjee](#):
>
> ```auto
> filter {
> if "Test-Info" in [message] {
> kv {
> source => "message"
> field_split => ", "
> value_split => "="
> trim_key => " "
> trim_value => " "
> prefix => ""
> }
> }
> }
> 
> ```

Very interesting!  
But I only want kv to process the part of the message that is after "Test-Info:[" and until "]".  
Because this message could have more data (before and after). Is there a way to limit kv only to this part and for the rest continue with something else (Eg grok?)

---

<div class="post-metadata">

**Author:** ![Mark\_S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_s/32/98686_2.png) [@Mark\_S](https://discuss.elastic.co/u/Mark_S)\
**Post date:** [March 11, 2023, 7:38am UTC](https://discuss.elastic.co/t/grok-or-any-alternative-to-search-for-keywords-in-logs/327351/7 "2023-03-11T07:38:42Z")

</div>

there could be a) extra spaces inside the Test-Info part and also b) more data before and after Test-Info that kv cannot handle - so also after kv, I need to continue processing with something else (eg grok). Kv must process only the part inside Test-Info:[kv-data].  
For example notice the extra space after protocol (in the other reply Sunile\_Manjee suggested to use trim\_value and trim\_key, but doesn't limit the processing within the first "[kv-data]" ) :

```auto
INFO ----Further.data-with-different-format ----WARN Test-Info:[protocol=HTTP/1.1 , uname=test-camel-svc, method=PUT, registert=02384827 , server=TESTVM9:8083, tracker_id=84d9d0231-2231-4a11-5e2d-88afa5ee12c6, bda_id=4352, testbda_id=?/?, cause=incoming_request, calling_server=XX.XXX.XXX.XXX, request_time=N/A, reception_date=2020-01-00T01:00:00.000+0000, time_elapsed_ms=37] 3123344 --- [XNIO-1 task-1] g.u.m.commons.logging.MDCLoggingFilter : processing_end, processing_end - more extra data;separatedwithother-symbols

```

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [March 11, 2023, 10:28am UTC](https://discuss.elastic.co/t/grok-or-any-alternative-to-search-for-keywords-in-logs/327351/8 "2023-03-11T10:28:14Z")

</div>

If you have variable fields, then you have to use Grok:

```auto
 grok {
       match => { "message" => "%{DATA}:%{SPACE}\[%{DATA:msg}\]%{SPACE}%{POSINT}%{SPACE}%{GREEDYDATA}" }

  }

```

---

<div class="post-metadata">

**Author:** ![Mark\_S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_s/32/98686_2.png) [@Mark\_S](https://discuss.elastic.co/u/Mark_S)\
**Post date:** [March 11, 2023, 10:41am UTC](https://discuss.elastic.co/t/grok-or-any-alternative-to-search-for-keywords-in-logs/327351/9 "2023-03-11T10:41:00Z")

</div>

Yes but for my case perhaps I could use first an if statement to see if Test-Info is included. If true, then use kv to get the key-value pairs. If not then continue with grok.  
Is this ok with logstash?

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [March 11, 2023, 12:18pm UTC](https://discuss.elastic.co/t/grok-or-any-alternative-to-search-for-keywords-in-logs/327351/10 "2023-03-11T12:18:23Z")

</div>

Whatever you want.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [March 11, 2023, 2:59pm UTC](https://discuss.elastic.co/t/grok-or-any-alternative-to-search-for-keywords-in-logs/327351/11 "2023-03-11T14:59:56Z")

</div>

> [@Mark\_S](#):
>
> `Test-Info:[protocol=HTTP/1.1 , uname=test-camel-svc, method=PUT, registert=02384827 , server=TESTVM9:8083, tracker_id=84d9d0231-2231-4a11-5e2d-88afa5ee12c6, bda_id=4352, testbda_id=?/?, cause=incoming_request, calling_server=XX.XXX.XXX.XXX, request_time=N/A, reception_date=2020-01-00T01:00:00.000+0000, time_elapsed_ms=37]`

If the extra space is on those cases you can use a `mutate` filter in the `kvMsg` field before the `kv` filter.

Something like this:

```auto
mutate {
    gsub => ["kvMsg"," , ",", "]
}

```

This will turn all the insaces where you have `<SPACE>,<SPACE>` into `,<SPACE>` and avoid the need to trim the fields individually.

---

<div class="post-metadata">

**Author:** ![Sunile\_Manjee](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sunile_manjee/32/111461_2.png) [@Sunile\_Manjee](https://discuss.elastic.co/u/Sunile_Manjee)\
**Post date:** [March 11, 2023, 3:52pm UTC](https://discuss.elastic.co/t/grok-or-any-alternative-to-search-for-keywords-in-logs/327351/12 "2023-03-11T15:52:26Z")

</div>

> [@Mark\_S](#):
>
> But I only want kv to process the part of the message that is after "Test-Info:[" and until "]".

can you try this which should only fetch KVs between `Test-Info:[` and `]`

```auto
filter {
  if "Test-Info" in [message] {
    grok {
      match => {
        "message" => "Test-Info:\[%{GREEDYDATA:test_info}\]"
      }
    }
    kv {
      source => "test_info"
      field_split => ", "
      value_split => "="
      trim_key => " "
      trim_value => " "
      prefix => ""
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![Mark\_S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_s/32/98686_2.png) [@Mark\_S](https://discuss.elastic.co/u/Mark_S)\
**Post date:** [March 12, 2023, 9:52am UTC](https://discuss.elastic.co/t/grok-or-any-alternative-to-search-for-keywords-in-logs/327351/13 "2023-03-12T09:52:02Z")

</div>

> [@Sunile\_Manjee](#):
>
> `test_info`

Thank you for the answers, but for some reason it does not seem to work. I don't get any error messages. The test\_info is created and i can see it in kibana, but kv does not seem to work. The other solution that involves dissect seems to be working

---

<div class="post-metadata">

**Author:** ![Mark\_S](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_s/32/98686_2.png) [@Mark\_S](https://discuss.elastic.co/u/Mark_S)\
**Post date:** [March 12, 2023, 10:02am UTC](https://discuss.elastic.co/t/grok-or-any-alternative-to-search-for-keywords-in-logs/327351/14 "2023-03-12T10:02:27Z")

</div>

Thank you! This works perfectly!!  
One last question: is it possible somehow to avoid adding kvMsg in the final message (i.e use it temporarily and then delete this keyword?)

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [March 12, 2023, 12:34pm UTC](https://discuss.elastic.co/t/grok-or-any-alternative-to-search-for-keywords-in-logs/327351/15 "2023-03-12T12:34:26Z")

</div>

> [@Mark\_S](#):
>
> One last question: is it possible somehow to avoid adding kvMsg in the final message (i.e use it temporarily and then delete this keyword?)

You may add a `remove_field => ["kvMsg"]` in the `kv` filter to remove the field if the filter is successful.

```auto
    kv {
        source => "kvMsg"
        value_split => "="
        field_split => ", "
        remove_field => ["kvMsg"]
    }

```

Or you could use a `[@metadata]` field, that will not be present in the output, just replace `kvMsg` with `[@metadata][kvMsg]` for example.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 9, 2023, 12:34pm UTC](https://discuss.elastic.co/t/grok-or-any-alternative-to-search-for-keywords-in-logs/327351/16 "2023-04-09T12:34:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.