Logstash filter for filtering out specific words from message

Elastic Stack Logstash
1

Hello team,

I am looking for a filter to filter out based on the words in a log message. For example,
Log messages - This is a test message from server hosted in AWS and This is a test message coming from server hosted in AWS
Requirement - New field with value test message from server and test message coming from server. that is, filter out the content between words "test" and "server" and put it into a new field.

I tried grok and dissect filter, but neither met my requirements.

many thanks,
Danish

2

What did you try? Both of those filters can be used to do this.

3

Grok pattern: This is a %{EXTRACT:program} hosted in AWS

Custom pattern: EXTRACT (\btest .* server\b)

Tested in grokdebug

4

Thank you for your prompt response, Rios. Could you please explain how to use that grok filter to filter out messages that contain two known words?

	2022-06-23T07:13:08.091Z	ERROR	controllers.IPpool	TEST-ERROR - IP inconsistency detected	{"namespace": "smoke-test", "name": "ip-out-of-range-test-ingress", "host": "ip-out-of-range-test.kubernetes-test.example.com", "frontend-ip": "0.0.0.0", "error": "Assigned IP outside of pool range."}
github.com/go-logr/zapr.(*zapLogger).Error
	/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128

I need to move the content between "TEST-ERROR" and "go-logr · GitHub" to a new field, regardless of what comes before or after that. This is how the new field should look:

IP inconsistency detected {"namespace": "smoke-test", "name": "ip-out-of-range-test-ingress", "host": "ip-out-of-range-test.kubernetes-test.example.com", "frontend-ip": "0.0.0.0", "error": "Assigned IP outside of pool range."}

I greatly appreciate your assistance with this.

Many thanks,
Danish

5

Hi Badger,

I couldn't figure out how to use apt dissect or grok filter between two words. Do you have any examples of how this might be used?

Many thanks,
Danish

6

You could use

grok { match => { "message" => "TEST-ERROR%{GREEDYDATA:someField}github.com/go-logr" } }

or

dissect { mapping => { "message" => "%{}TEST-ERROR%{someField}github.com/go-logr%{}"

The latter is probably cheaper.

2 Likes
7

Thank you very much, Badger, that worked!! I greatly appreciate your assistance with this.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.