Grok filter for email address alteration

Elastic Stack Beats
1

Hi,

Am trying to pre-filter some data in Filebeat, i have a issue i cannot resolve from picking up some weird logs.

I'm trying to pickup an email address from the file as well as an ip and date time etc... whilst it all works what i have an problem with is the email addresses.

they are all prefixed with characters i want to remove... i.e ul2007alfie@alien.net and its the ul2007 i need to remove.

I'm not particularly skilled with grok, and i have been trying to research this all day and have been having many adventures on https://grokdebug.herokuapp.com/

So my filter i have been testing in herouapp is this

%{IPORHOST}%{NOTSPACE}%{WORD:dummy}%{NOTSPACE}%{MONTH} %{MONTHDAY}, %{YEAR} %{TIME} %{WORD}

(details changed to protect the strange and odd )
and the log line i'm trying to parse is:

["1.1.1.1","\ul2007alfie@alien.net\ul2007","AWS Directory Service Reset Password Request","dummy","Sep 25, 2019 4:33:51 "

IP: 1.1.1.1
email address: alfie@alien.net
subject: AWS Directory Service Reset Password Request
user: dummy
Date: Sep 25, 2019 4:33:51

so i have tried various permutations and whilst i can omit the ul2007 from the beginning, the one at the end remains and instead of seeing alfie@alien.net\ul2007 it gets separated out by fields... so im not sure if the grokdebugger is 100% effective 100% of the time as i have been getting weird results.

So i tried to mutate using gsub, but i'm not sure of the format for this.

mutate {
gsub => [
"message", "ul2007"
%{NOTSPACE:email_address}
]
}

This dosn't work, as it reports no matches when run in the debugger. I've been searching through the forums and google for anything close to what might hint at a way forward, but i feel i'm going around in circles

Can anyone give me some pointers on what i should be looking for? is gsub the way to go and mutate the data into a new field?

2

Still having fun in filter land....

tried these variations and also read up on KV as well but i don't think that would work here as i havn't yet declared a key ?

filter {
mutate {
gsub => [
"%{NOTSPACE:email_address}", "u0027]", "",
]
}
}

filter {
mutate {
gsub => [ "message", "u0027", "" ]
}
}

%{NOTSPACE:email_address}
filter {
mutate {
gsub => [ "message", "u0027", "" ]
}
}

%{GREEDYDATA}
filter {
mutate {
gsub => [
"%{NOTSPACE:email_address}", "u0027", "",
]
}
}

%{NOTSPACE:email_address}
filter {
kv {
trim => "u0027,"
}
}

with these i get no match, so i guess its either parsing the whole line and not matching or its not reading the line at all.
i removed everything and am just using the u0027alfie@alien.net so as to not complicate things.
Thing is, when im reading through the help sections that describe what filters do, how they work and how they are structured with the examples, they just don't seem to work, even with simple data like the malformed email address.

so let me try and get this straight..

%{NOTSPACE:email_address}

reads the data, shows me the output in the debug window.. great, parses the data, so now i need to alter the data in the field and then pass the altered data back to the field.

is that how this should be structured ?

3

i think i have it

\u0027%{NOTSPACE:email_address}\u0027

is the correct line to use for this. i'm going to try this now in my main filter and see how it goes.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.