Logstash replace string problem

Elastic Stack Logstash
1

Hi

I'm new to the logstash and don't know if what I want to do is even possible. I need to replace some information in logs provided by an application. My example input is:
2017-09-18 10:45:25,404 ERROR some variable text 10.23.42.45 some variable text 10.45.22.111
Task for me is to replace those IP's with other text. At first I tried to use below grok:

grok{
			match => {"log" => ["%{DATA}(?<ip>%{IPV4})%{GREEDYDATA}"]}
		}
		if "_grokparsefailure" not in [tags]{
			mutate{
				gsub => ["log","%{ip}","SOMEIP"]
			}
		}

But this way I'm only replacing the first IP and the second one stays unchanged as grok takes first match and moves further, I tried to use %{IPV4} pattern inside mutate but it doesn't work. The main problem is that logs that I have to parse aren't standardized and I have to search for those IP inside a message. Also the occurrence of those IP inside a message varies.

2

Hi @Arkadiusz,

first and important, play around with the grok debugger, its a powerfull tool for testing this kind of stuff :slight_smile:

Grok Debugger

then you should try something like this:

2017-09-18 10:45:25,404 ERROR some variable text 10.23.42.45 some variable text 10.45.22.111

%{TIMESTAMP_ISO8601}%{DATA}%{IPV4}%{DATA}%{IPV4}

just as an example.

if [IPV4] {
   mutate...
  }

if ![IPV4]
looks for "not exists"

if [IPV4]
looks for "exists"

Cheers,
Dirk

3

Hi @lueneburger
Thank you for fast response. I was using grok debugger but like I wrote the logs aren't standardized and this was only an example to show what I want to accomplish. And as the occurrence of the IP and the number of times it occurs varies so my Grok Pattern would have to look more like this:

"%{DATA}(?<ip1>%{IPV4})%{DATA}(?<ip2>%{IPV4})%{DATA}(?<ip3>%{IPV4})%{DATA}(?<ip4>%{IPV4})%{DATA}"

Which I wanted to avoid, and what if there will be 5th IP etc. Best solution would be to use a loop but from what I found out there is no for statement for logstash.

4

The gsub function of the mutate filter might help, depending on what text you want to replace the IPs with.

5

Hi
Yeah I have tried this but the problem is that IP's are different and mutate gsub replace one string with another, and when i tried to use pattern %{IPV4} with mutate it didn't worked.

6

There's no need to use grok to locate the exact IP address first. Just use gsub:

mutate {
  gsub => ["message", "\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b", "FOO"]
}

(You could use a more exact regexp to match an IP address but this one's probably good enough.)

7

Hi @magnusbaeck
Thank you this is exactly what I was looking for.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.