Hi,
Am trying to pre-filter some data in Filebeat, i have a issue i cannot resolve from picking up some weird logs.
I'm trying to pickup an email address from the file as well as an ip and date time etc... whilst it all works what i have an problem with is the email addresses.
they are all prefixed with characters i want to remove... i.e ul2007alfie@alien.net and its the ul2007 i need to remove.
I'm not particularly skilled with grok, and i have been trying to research this all day and have been having many adventures on https://grokdebug.herokuapp.com/
So my filter i have been testing in herouapp is this
%{IPORHOST}%{NOTSPACE}%{WORD:dummy}%{NOTSPACE}%{MONTH} %{MONTHDAY}, %{YEAR} %{TIME} %{WORD}
(details changed to protect the strange and odd )
and the log line i'm trying to parse is:
["1.1.1.1","\ul2007alfie@alien.net\ul2007","AWS Directory Service Reset Password Request","dummy","Sep 25, 2019 4:33:51 "
IP: 1.1.1.1
email address: alfie@alien.net
subject: AWS Directory Service Reset Password Request
user: dummy
Date: Sep 25, 2019 4:33:51
so i have tried various permutations and whilst i can omit the ul2007 from the beginning, the one at the end remains and instead of seeing alfie@alien.net\ul2007 it gets separated out by fields... so im not sure if the grokdebugger is 100% effective 100% of the time as i have been getting weird results.
So i tried to mutate using gsub, but i'm not sure of the format for this.
mutate {
gsub => [
"message", "ul2007"
%{NOTSPACE:email_address}
]
}
This dosn't work, as it reports no matches when run in the debugger. I've been searching through the forums and google for anything close to what might hint at a way forward, but i feel i'm going around in circles
Can anyone give me some pointers on what i should be looking for? is gsub the way to go and mutate the data into a new field?