Email grok filter logstash

rouchad_rouchad · May 28, 2018, 2:40pm

hello everyone, I want your help please
I connected my mailbox to elasticsearch and made filters on messages that come from some senders using : grok debugger
what i want is how to modify my code logstash .the filters are ready just i want hte way to put them in my logstash config for example:
if I receive a message from aaaa@gmail.com
here is the filter I will apply: XXXXXX

if I receive an email from bbbbb@gmail.com
here is the filter that I will apply
ie I want to make conditions inside my logstash code
thank you

Badger · May 28, 2018, 4:24pm

There is a section on conditionals, including examples, in the documentation.

rouchad_rouchad · May 29, 2018, 3:35pm

thank u very much Badger
can u help me please ? there is an error in my logstash config and i don't know how to execute it :

input {
imap {
host => "imap.gmail.com"
password => "XXXXXXXXX"
user => "rouchad767@gmail.com"
port => 993
secure => true
fetch_count => 15
check_interval => 10
strip_attachments => true
folder => "Inbox"
}
}
filter {
grok {
if [from] == "abuseorange47@gmail.com"
match => { "message" => "%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{WORD}%{NOTSPACE}%{SPACE}%{GREEDYDATA:receiver}
%{GREEDYDATA:message}
%{WORD}%{NOTSPACE}%{SPACE}%{GREEDYDATA:time_sent}
%{GREEDYDATA:message}
%{CISCO_REASON}%{NOTSPACE}%{SPACE}%{GREEDYDATA:ID}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:localisation}
%{GREEDYDATA:localisation}
%{GREEDYDATA:localisation}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{CISCO_REASON}%{NOTSPACE}%{SPACE}%{GREEDYDATA:name_of_movie}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:name_Director}
%{GREEDYDATA:mission_of_director}
%{GREEDYDATA:security_company}
%{WORD}%{NOTSPACE}%{SPACE}%{GREEDYDATA:sender}
%{WORD}%{NOTSPACE}%{SPACE}%{GREEDYDATA:address}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{WORD}%{NOTSPACE}%{SPACE}%{GREEDYDATA:title_movie}
%{CISCO_REASON}%{NOTSPACE}%{SPACE}%{GREEDYDATA:attack_time}
%{CISCO_REASON}%{NOTSPACE}%{SPACE}%{IP:IP_Adress}
%{WORD}%{NOTSPACE}%{SPACE}%{GREEDYDATA:port}
%{WORD}%{NOTSPACE}%{SPACE}%{GREEDYDATA:Attack_type}
%{CISCO_REASON}%{NOTSPACE}%{SPACE}%{GREEDYDATA:hash}
%{WORD}%{NOTSPACE}%{SPACE}%{GREEDYDATA:file_name}
%{WORD}%{NOTSPACE}%{SPACE}%{GREEDYDATA:file_size}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{SPACE}%{GREEDYDATA:Victim_Company}
%{SPACE}%{GREEDYDATA:Security_Company}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{SPACE}%{GREEDYDATA:phone}
%{SPACE}%{GREEDYDATA:email_security_company}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{SPACE}%{GREEDYDATA:receiver_Company}
%{SPACE}%{GREEDYDATA:email_receiver_company}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}
%{GREEDYDATA:message}" }
}
}
output {
stdout { codec => rubydebug }
elasticsearch {
index => "InterAbuse "
document_type => "email"
hosts => "localhost:9200"
}

}

Badger · May 29, 2018, 3:59pm

You cannot place a conditional inside a filter (or input or output). So you need to reverse the order of those two lines, add a { at the end of the if, and add a } before output {.

Once you get that fixed you will have a problem with the grok, probably a timeout. As a rule of thumb, you should only have one DATA or GREEDYDATA in a pattern. 91 is not going to work. Are the emails really all in a very specific 92 line pattern?

rouchad_rouchad · May 29, 2018, 4:10pm

thanks a lot for your help, yes I tested the filter in grok debugger and it worked well
I took the message that I put it in grok debugger I applied the filter and I found the good result

if u took this email and u put it with the filter in grok debugger it will work very nice

there is the email :
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

To: nocisp.oma@orange.com

DATE: 2018-03-07

CASE ID: c815ea60980b2a3b3d17

Dear Meditelecom:

We are contacting you on behalf of The Cartoon Network, Inc. with location at:

The Cartoon Network, Inc.
1050 Techwood Drive, NW
Atlanta GA 30318

Under penalty of perjury we assert that IP-Echelon Pty. Ltd. is authorized to act on behalf of The Cartoon Network, Inc., who is the copyright owner and/or owner of exclusive rights in such content identified below.

We have become aware that an individual has utilized the IP address 196.118.167.130 at the recorded date and time below to download, host, and/or facilitate the downloading and/or streaming of video content that is exclusively owned by The Cartoon Network, Inc.. Such unauthorized distribution of The Cartoon Network, Inc. content without the express written authorization of The Cartoon Network, Inc. constitutes copyright infringement. This conduct may also violate the laws of other countries, international law, and/or treaty obligations.

We have a good faith belief that the use of the copyrighted material identified below is not authorized by The Cartoon Network, Inc., its agent, or the law.

The title in question: Rick and Morty

As the owner of the IP address, we request that you immediately assist in removing and disabling access to the infringing material from your network. Additionally we request that you contact the subscriber who has engaged in the conduct described above and take further steps to prevent the subscriber from unauthorized future use and sharing of The Cartoon Network, Inc. content.

We state, under penalty of perjury, that the information in this notification is accurate and that we are authorized to act on behalf of the owner of the exclusive right that is allegedly infringed herein.

Nothing in this notice shall operate as a waiver of any rights, claims, or remedies available to The Cartoon Network, Inc. with respect to the infringement alleged herein. All such rights, claims, and remedies are hereby expressly reserved.

We appreciate your attention to and cooperation with this notice. Please provide a prompt response specifying actions you have taken to resolve this matter.

Do not hesitate to contact me with any questions. You may reach me via email at p2p@copyright.ip-echelon.com.

Regards,

Adrian Leatherland
CEO
IP-Echelon
Email: p2p@copyright.ip-echelon.com
Address: 7083 Hollywood Blvd., Los Angeles, CA 90028, United States

------------- Infringement Details ----------------------------------
Title: Rick and Morty
Timestamp: 2018-03-07T19:43:28Z
IP Address: 196.118.167.130
Port: 28159
Type: BitTorrent
Torrent Hash: 3bd437314a23e67e2845c1ca1a1f652a90d3ddcd
Filename: Rick.and.Morty.S03E01.HDTV.x264-W4F[eztv].mkv
Filesize: 161 MB

c815ea60980b2a3b3d17
Open
Normal


The Cartoon Network, Inc.
IP-Echelon - Compliance

6715 Hollywood Blvd
Los Angeles CA 90028
United States of America

+1 (310) 606 2747
p2p@copyright.ip-echelon.com


Meditelecom
nocisp.oma@orange.com


2018-03-07T19:43:28Z
196.118.167.130
28159
BitTorrent

1



  2018-03-07T19:43:28Z
  
  Rick.and.Morty.S03E01.HDTV.x264-W4F[eztv].mkv
  169720295
  3bd437314a23e67e2845c1ca1a1f652a90d3ddcd

system · June 26, 2018, 4:10pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.