Index matched patterns and drop unmatched

Hi. I'm new to Logstash grok patterns and don't have enough knowledge to right the grok filter to match and drop events.

I'm trying to index events where the pattern matches event_type = tripwire write to tripwire index, eventype with wineventlog to windeventlog index and remaining events to drop.

Raw Events from text file(replaced actual hostnames and Ip address)

{"event_type":"tripwire","raw_message":"<134>May 06 20:08:09 5.0.193.55 TE: Information HostName=abcde LogId=-1y2p0ij32e8d4:-xxxxxxx LogCategory=System LogUser=system NodeId=-xxxxxx NodeName=xxxxxx NodeIp=xxxxxxxx AssociatedObjects=-1y2p0ij32e8bw:-1y2p0iejzsjfs Msg="TE agent Windows Server started." ","month":"05","forwarder":abcd111","@timestamp":"2019-05-07T01:31:24.127Z","priority":"134","time_generated":"2019-05-07T01:08:10.702451+00:00","year":"2019","day":"07","fromhost_name":","facility":"16","time_reported":"2019-05-06T20:08:09+00:00","@version":"1","severity":"6","fromhost_ip":"x.x.x.x."}
{"event_type":"wineventlog","raw_message":"SPL_WIN-SECURITY: ABCD1234 05/06/2019 08:08:06 PM LogName=Security SourceName=Microsoft Windows security auditing. EventCode=4624 EventType=0 Type=Information ComputerName=ABCD1234 TaskCategory=Logon OpCode=Info RecordNumber=18 Keywords=Audit Success Message=An account was successfully logged on. Subject: Security ID: S-1-0-0 Account Name: - Account Domain: - Logon ID: 0x0 Logon Type: 3 Impersonation Level: Impersonation New Logon: Security ID: S-1-5-21 Account Name: 70 Account Domain: CCC Logon ID: 0x32793167 Logon GUID: Process Information: Process ID: 0x0 Process Name: - Network Information: Workstation Name: - Source Network Address: 11.11.222.172 Source Port: 58355 Detailed Authentication Information: Logon Process: Kerberos Authentication Package: Kerberos Transited Services: - Package Name (NTLM only): - Key Length: 0 This event is generated when a logon session is created. It is generated on the computer that was accessed. The subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe. The logon type field indicates the kind of logon that occurred. The most common types are 2 (interactive) and 3 (network). The New Logon fields indicate the account for whom the new logon was created, i.e. the account that was logged on. The network fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases. The impersonation level field indicates the extent to which a process in the logon session can impersonate. The authentication information fields provide detailed information about this specific logon request. - Logon GUID is a unique identifier that can be used to correlate this event with a KDC event. - Transited services indicate which intermediate services have participated in this logon request. - Package name indicates which sub-protocol was used among the NTLM protocols. - Key length indicates the length of the generated session key. This will be 0 if no session key was requested.","month":"05","forwarder":.abcd111","@timestamp":"2019-05-07T01:31:24.152Z","priority":"13","time_generated":"2019-05-07T01:08:10.704045+00:00","year":"2019","day":"07","fromhost_name":"localhost","facility":"1","time_reported":"2019-05-07T01:08:10.704045+00:00","@version":"1","severity":"5","fromhost_ip":"127.0.0.1"}
{"event_type":"cisco-asa","raw_message":"<166>May 07 2019 01:13:06 30.31.80.15 : %ASA-6-302014: Teardown TCP connection 215574701 for v910:xx.xx.xx.xx/61995(LOCAL\111111) to v980:xx.x.x.x/443 duration 0:00:00 bytes 10366 TCP FINs","month":"05","forwarder":abcd111","@timestamp":"2019-05-07T01:31:24.158Z","priority":"166","time_generated":"2019-05-07T01:13:06.862205+00:00","year":"2019","day":"07","fromhost_name":"xx.xx.xx.xx","facility":"20","time_reported":"2019-05-07T01:13:06+00:00","@version":"1","severity":"6","fromhost_ip":"xx.xx.xx.xx"}

This is how my logstash.conf look currently, I'm sure it's not right :slight_smile:

input {
file {
path => "/tmp/topic_message.log"
type => "log"
codec => json {
pattern => "^{"event_type":"tripwire","raw_message":"<134>%{DATA:timestamp} Information
}
}

filter {
if "_grokparsefailure" in [tags] {
drop { }
}
}

output {
stdout {codec => rubydebug}
if [type] == "tripwire" {
elasticsearch {
hosts => "localhost:9200"
index => "tripwire"
}
}
if [type] == "wineventlog" {
elasticsearch {
hosts => "localhost:9200"
index => "wineventlog"
}
}
else {
elasticsearch {
hosts => "localhost:9200"
index => "test"
}
}
}

Please someone help

You have mangled all those lines so that they are no longer valid JSON. Quotes are not escaped, or missing and so on. Also you need to block quote the JSON. Edit your post, select the three lines and click on </> in the toolbar above the edit pane.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.