Split long log into small parts and apply grok in logstash filter


(Tharaka) #1

I have a log which cannot directly extract using grok patterns under logstash filter. I have tested full grok pattern for the log in grok debugger and it works fine. But when I run the grok pattern under logstash filter gives grok failure error.

So I need to break full message into small parts and apply the grok individually.

I need to know, whether above mention requirement is possible under logstash ?

If yes, need a guide line to perform.


(Colton) #2

Do you know why the log is being marked as grokparse failure? Maybe we could help you solve that issue.

If you are dead set on splitting up your log, how are you reading the log in? Through a file? Syslog? If you are reading a the log in using the file input that has a configurable option called delimiter which allows you to set a new line delimiter.

Perhaps you could attach a sample log, the grok you constructed and your logstash.conf file, removing any sensitive information.


(Tharaka) #3

@CDR

Below is the grok patterns put under logstash filter.

filter {

grok {
	match => [ "message", "<%{POSINT:pri_id}>%{SYSLOGTIMESTAMP:log_timestamp} %{HOSTNAME:hostname} %{WORD:source}: \[%{DATA:num}\] %{GREEDYDATA:signature} \[Impact: %{DATA:impact}\] From \\"%{DATA:device}\\" %{WORD:seq} %{WORD:day} %{SYSLOGTIMESTAMP:trigger_timestamp} %{DATA:list_year} %{WORD:time_zone} \[Classification: %{GREEDYDATA:classification}\] \[Priority: %{NUMBER:priority}\] \{%{DATA:protocol}\} (?<srcip>[0-9]+.[0-9]+.[0-9]+.[0-9]+|N/A):(?<srcport>[0-9]+|N/A) \(%{DATA:srcname}\)->(?<dstip>[0-9]+.[0-9]+.[0-9]+.[0-9]+|N/A):(?<dstport>[0-9]+|N/A) \(%{DATA:dstname}\)"]
  }	
}

and here are the sample log. Some sensitive information are masked here.

<46>Oct 10 11:44:29 XXX-XX-XX XXXXS: [135:2:2] internal: session established [Impact: Potentially Vulnerable] From "XXX-XX-XX" at Tue Oct 10 11:44:29 2017 UTC [Classification: A TCP Connection was Detected] [Priority: 3] {tcp} 192.168.1.1:23 (unknown)->10.10.10.10:XXXXX (unknown)

<46>Oct 10 11:44:32 XXX-XX-XX XXXXS: [119:4:1] http_inspect: BARE BYTE UNICODE ENCODING [Impact: Potentially Vulnerable] From "XXX-XX-XX" at Tue Oct 10 11:44:31 2017 UTC [Classification: Not Suspicious Traffic] [Priority: 3] {tcp} 10.10.10.XXX:XXXXX (unknown)->192.168.1.1:80 (unknown)"


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.