Logstash filter for filtering out specific words from message

Hello team,

I am looking for a filter to filter out based on the words in a log message. For example,
Log messages - This is a test message from server hosted in AWS and This is a test message coming from server hosted in AWS
Requirement - New field with value test message from server and test message coming from server. that is, filter out the content between words "test" and "server" and put it into a new field.

I tried grok and dissect filter, but neither met my requirements.

many thanks,
Danish

What did you try? Both of those filters can be used to do this.

Grok pattern: This is a %{EXTRACT:program} hosted in AWS

Custom pattern: EXTRACT (\btest .* server\b)

Tested in grokdebug

Thank you for your prompt response, Rios. Could you please explain how to use that grok filter to filter out messages that contain two known words?

	2022-06-23T07:13:08.091Z	ERROR	controllers.IPpool	TEST-ERROR - IP inconsistency detected	{"namespace": "smoke-test", "name": "ip-out-of-range-test-ingress", "host": "ip-out-of-range-test.kubernetes-test.example.com", "frontend-ip": "0.0.0.0", "error": "Assigned IP outside of pool range."}
github.com/go-logr/zapr.(*zapLogger).Error
	/go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128

I need to move the content between "TEST-ERROR" and "go-logr · GitHub" to a new field, regardless of what comes before or after that. This is how the new field should look:

IP inconsistency detected {"namespace": "smoke-test", "name": "ip-out-of-range-test-ingress", "host": "ip-out-of-range-test.kubernetes-test.example.com", "frontend-ip": "0.0.0.0", "error": "Assigned IP outside of pool range."}

I greatly appreciate your assistance with this.

Many thanks,
Danish

Hi Badger,

I couldn't figure out how to use apt dissect or grok filter between two words. Do you have any examples of how this might be used?

Many thanks,
Danish

You could use

grok { match => { "message" => "TEST-ERROR%{GREEDYDATA:someField}github.com/go-logr" } }

or

dissect { mapping => { "message" => "%{}TEST-ERROR%{someField}github.com/go-logr%{}"

The latter is probably cheaper.

2 Likes

Thank you very much, Badger, that worked!! I greatly appreciate your assistance with this.