Filter grok logstash email failed

Elastic Stack Logstash
1

hello everyone please I need your help!
I connected my mailbox to elasticsearch using imap
the problem is that I want to apply grok type filters on
I want to make condtions in my logstash config ie filters type grok ===> if the email comes from this address here is the filter that I will apply to this email

if the email comes from this address here is the filter that I want to apply etc
I made an attempt but I failed ... when I apply the filter I can not find anything at kibana I find the normal message as if I did not apply filters

there is my config file :

input {
imap {
host => "imap.gmail.com"
password => "XXXXXXX"
user => "rouchad767@gmail.com"
port => 993
secure => true
fetch_count => 15
check_interval => 10
strip_attachments => true
folder => "Inbox"
}
}
filter {
if [from] == "abuseorange47@gmail.com" {
grok {
match => { "message" => "%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{WORD}%{NOTSPACE}%{SPACE}%{GREEDYDATA:receiver}
%{GREEDYDATA:message}
%{WORD}%{NOTSPACE}%{SPACE}%{GREEDYDATA:time_sent}
%{GREEDYDATA:message}
%{CISCO_REASON}%{NOTSPACE}%{SPACE}%{GREEDYDATA:ID}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:localisation}
%{GREEDYDATA:localisation}
%{GREEDYDATA:localisation}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{CISCO_REASON}%{NOTSPACE}%{SPACE}%{GREEDYDATA:name_of_movie}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:name_Director}
%{GREEDYDATA:mission_of_director}
%{WORD}%{NOTSPACE}%{SPACE}%{GREEDYDATA:sender}
%{WORD}%{NOTSPACE}%{SPACE}%{GREEDYDATA:address}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{WORD}%{NOTSPACE}%{SPACE}%{GREEDYDATA:title_movie}
%{CISCO_REASON}%{NOTSPACE}%{SPACE}%{GREEDYDATA:attack_time}
%{CISCO_REASON}%{NOTSPACE}%{SPACE}%{IP:IP_Adress}
%{WORD}%{NOTSPACE}%{SPACE}%{GREEDYDATA:port}
%{WORD}%{NOTSPACE}%{SPACE}%{GREEDYDATA:Attack_type}
%{CISCO_REASON}%{NOTSPACE}%{SPACE}%{GREEDYDATA:hash}
%{WORD}%{NOTSPACE}%{SPACE}%{GREEDYDATA:file_name}
%{WORD}%{NOTSPACE}%{SPACE}%{GREEDYDATA:file_size}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{SPACE}%{GREEDYDATA:Victim_Company}
%{SPACE}%{GREEDYDATA:Security_Company}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{SPACE}%{GREEDYDATA:phone}
%{SPACE}%{GREEDYDATA:email_security_company}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{SPACE}%{GREEDYDATA:receiver_Company}
%{SPACE}%{GREEDYDATA:email_receiver_company}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}" }
}
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
index => "data"
document_type => "email"
hosts => "localhost:9200"
}

}

what i need to do ! please help me

2

How could we possibly help out if we don't know what an example email looks like? Copy/paste what your stdout { codec => rubydebug } output produces.

1 Like
3

i apply the grok filter and there is what i get in kibana the same email i have in my mail box :
The title in question: Rick and Morty

As the owner of the IP address, we request that you immediately assist in removing and disabling access to the infringing material from your network. Additionally we request that you contact the subscriber who has engaged in the conduct described above and take further steps to prevent the subscriber from unauthorized future use and sharing of The Cartoon Network, Inc. content.

We state, under penalty of perjury, that the information in this notification is accurate and that we are authorized to act on behalf of the owner of the exclusive right that is allegedly infringed herein.

Nothing in this notice shall operate as a waiver of any rights, claims, or remedies available to The Cartoon Network, Inc. with respect to the infringement alleged herein. All such rights, claims, and remedies are hereby expressly reserved.

We appreciate your attention to and cooperation with this notice. Please provide a prompt response specifying actions you have taken to resolve this matter.

Do not hesitate to contact me with any questions. You may reach me via email at p2p@copyright.ip-echelon.com.

Regards,

Adrian Leatherland
CEO
IP-Echelon
Email: p2p@copyright.ip-echelon.com
Address: 7083 Hollywood Blvd., Los Angeles, CA 90028, United States

  • ------------- Infringement Details ----------------------------------
    Title: Rick and Morty
    Timestamp: 2018-03-07T19:43:28Z
    IP Address: 196.118.167.130
    Port: 28159
    Type: BitTorrent
    Torrent Hash: 3bd437314a23e67e2845c1ca1a1f652a90d3ddcd
    Filename: Rick.and.Morty.S03E01.HDTV.x264-W4F[eztv].mkv
    Filesize: 161 MB

and what i want to see in kibana after apply the filters :
{
"CISCO_REASON": [
[
"The title in question",
"Timestamp",
"IP Address",
"Torrent Hash"
]
],
"WORD": [
[
"question",
"Email",
"Address",
"Title",
"Timestamp",
"Address",
"Port",
"Type",
"Hash",
"Filename",
"Filesize"
]
],
"NOTSPACE": [
[
":",
":",
":",
":",
":",
":",
":",
":",
":",
":",
":"
]
],
"SPACE": [
[
" ",
" ",
" ",
" ",
" ",
" ",
" ",
" ",
" ",
" ",
" "
]
],
"name_of_movie": [
[
"Rick and Morty"
]
],
"message": [
[
"",
"As the owner of the IP address, we request that you immediately assist in removing and disabling access to the infringing material from your network. Additionally we request that you contact the subscriber who has engaged in the conduct described above and take further steps to prevent the subscriber from unauthorized future use and sharing of The Cartoon Network, Inc. content.",
"",
"We state, under penalty of perjury, that the information in this notification is accurate and that we are authorized to act on behalf of the owner of the exclusive right that is allegedly infringed herein.",
"",
"Nothing in this notice shall operate as a waiver of any rights, claims, or remedies available to The Cartoon Network, Inc. with respect to the infringement alleged herein. All such rights, claims, and remedies are hereby expressly reserved.",
"",
"We appreciate your attention to and cooperation with this notice. Please provide a prompt response specifying actions you have taken to resolve this matter.",
"",
"Do not hesitate to contact me with any questions. You may reach me via email at p2p@copyright.ip-echelon.com.",
"",
"Regards,",
"",
"",
"- ------------- Infringement Details ----------------------------------",
"- ---------------------------------------------------------------------"
]
],
"name_Director": [
[
"Adrian Leatherland"
]
],
"mission_of_director": [
[
"CEO"
]
],
"security_company": [
[
"IP-Echelon"
]
],
"sender": [
[
"p2p@copyright.ip-echelon.com"
]
],
"address": [
[
"7083 Hollywood Blvd., Los Angeles, CA 90028, United States"
]
],
"title_movie": [
[
"Rick and Morty"
]
],
"attack_time": [
[
"2018-03-07T19:43:28Z"
]
],
"IP_Adress": [
[
"196.118.167.130"
]
],
"IPV6": [
[
null
]
],
"IPV4": [
[
"196.118.167.130"
]
],
"port": [
[
"28159"
]
],
"Attack_type": [
[
"BitTorrent"
]
],
"hash": [
[
"3bd437314a23e67e2845c1ca1a1f652a90d3ddcd"
]
],
"file_name": [
[
"Rick.and.Morty.S03E01.HDTV.x264-W4F[eztv].mkv"
]
],
"file_size": [
[
"161 MB"
]
]
}

4

Good Morning Magnus

here is another example of an email that I want to deal with with grok filter ..
the problem is that i can not apply the filter correctly i tested the filter in grok debugger and it worked well but when i run my logstash file and i see the email in kibana nothing has changed as if I did not do filters

there is the email :

Dear Sir/Madam,

We have detected abuse from the IP address ( 197.230.107.154 ), which according to a whois lookup is on your network. We would appreciate if you would investigate and take action as appropriate. Any feedback is welcome but not mandatory.

Log lines are given below, but please ask if you require any further information.

(If you are not the correct person to contact about this please accept our apologies - your e-mail address was extracted from the whois record by an automated process. This mail was generated by Fail2Ban.)

IP of the attacker: 197.230.107.154

You can contact us by using: abuse-reply@keyweb.de

Addresses to send to:
noc_isp@meditel.ma

==================== Excerpt from log for 197.230.107.154 ====================
Note: Local timezone is +0100 (CET)
Mar 7 20:44:05 shared07 sshd[5371]: Invalid user admin from 197.230.107.154
Mar 7 20:44:05 shared07 sshd[5371]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=197.230.107.154
Mar 7 20:44:07 shared07 sshd[5371]: Failed password for invalid user admin from 197.230.107.154 port 33040 ssh2
Mar 7 20:44:08 shared07 sshd[5371]: Connection closed by 197.230.107.154 port 33040 [preauth]

how i want to see the email in kibana after apply the grok filter :
"message": [
[
"Dear Sir/Madam",
"We have detected abuse from the IP address ( 197.230.107.154 ), which according to a whois lookup is on your network. We would appreciate if you would investigate and take action as appropriate. Any feedback is welcome but not mandatory",
"Log lines are given below, but please ask if you require any further information",
"(If you are not the correct person to contact about this please accept our apologies - your e-mail address was extracted from the whois record by an automated process. This mail was generated by Fail2Ban.",
"Addresses to send to",
"Excerpt from log for 197.230.107.154 ===================",
"Note: Local timezone is +0100 (CET)
"Attacker_IP": [
[
"197.230.107.15"
]
],
"sender": [
[
"abuse-reply@keyweb.d"
]
],
"receiver": [
[
"noc_isp@meditel.m"
]
],
"time_attack": [
[
"Mar 7 20:44:05",
"Mar 7 20:44:05",
"Mar 7 20:44:07",
"Mar 7 20:44:08"
]
],
"Attack": [
[
"Invalid user admin from 197.230.107.15",
"pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=197.230.107.15",
"Failed password for invalid user admin from 197.230.107.154 port 33040 ssh",
"Connection closed by 197.230.107.154 port 33040 [preauth]"
]
]

there is my config file logstash :
input {
imap {
host => "imap.gmail.com"
password => "xxxxxxx"
user => "rouchad767@gmail.com"
port => 993
secure => true
fetch_count => 15
check_interval => 10
strip_attachments => true
folder => "Inbox"
}
}
filter {
if [from] == "abuseorange47@gmail.com" {
grok {
match => { "message" => "%{GREEDYDATA:message}%{NOTSPACE}%{SPACE}%{GREEDYDATA:message}%{NOTSPACE}%{SPACE}%{GREEDYDATA:message}%{NOTSPACE}%{SPACE}%{GREEDYDATA:message}%{NOTSPACE}%{SPACE}%{CISCO_REASON}%{NOTSPACE}%{SPACE}%{GREEDYDATA:Attacker_IP}%{NOTSPACE}%{SPACE}%{CISCO_REASON}%{NOTSPACE}%{SPACE}%{GREEDYDATA:sender}%{NOTSPACE}%{SPACE}%{GREEDYDATA:message}%{NOTSPACE}%{SPACE}%{GREEDYDATA:receiver}%{NOTSPACE}%{SPACE}%{NOTSPACE}%{SPACE}%{GREEDYDATA:message}%{NOTSPACE}%{SPACE}%{GREEDYDATA:message}%{SPACE}%{NOTSPACE}%{SPACE}%{CISCOTIMESTAMP:time_attack}%{SPACE}%{WORD}%{SPACE}%{WORD}%{SPACE}%{NOTSPACE}%{WORD}%{SPACE}%{NOTSPACE}%{SPACE}%{GREEDYDATA:Attack}%{NOTSPACE}%{SPACE}%{CISCOTIMESTAMP:time_attack}%{SPACE}%{WORD}%{SPACE}%{WORD}%{SPACE}%{NOTSPACE}%{WORD}%{SPACE}%{NOTSPACE}%{SPACE}%{GREEDYDATA:Attack}%{NOTSPACE}%{SPACE}%{CISCOTIMESTAMP:time_attack}%{SPACE}%{WORD}%{SPACE}%{WORD}%{SPACE}%{NOTSPACE}%{WORD}%{SPACE}%{NOTSPACE}%{SPACE}%{GREEDYDATA:Attack}%{NOTSPACE}%{SPACE}%{CISCOTIMESTAMP:time_attack}%{SPACE}%{WORD}%{SPACE}%{WORD}%{SPACE}%{NOTSPACE}%{WORD}%{SPACE}%{NOTSPACE}%{SPACE}%{GREEDYDATA:Attack}" }
}
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
index => "golden"
document_type => "email"
hosts => "localhost:9200"
}

}

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.