Need to create filter for logstash

[mlala (muntazir)]

Hi,

I am trying to prepare filter for F5 ASM but using GROK Debugger i am facing issue related to same can anyone help with with proper logstash filter. i will paste the log file for your convenience.

<134>Mar 8 12:40:06 DC-INT-SLB-02.zms.local ASM:"192.168.48.118","N/A","N/A","192.168.35.166","443","2018-03-08 12:40:06","Information Leakage","/Common/www.xxx.lm.sa.app/www.xxx.lm.sa_vs","192.168.48.118%0","","192.168.90.14","POST","2016-04-10 10:03:56","/Common/www.xxx.lm.sa.app/www.xxx.lm.sa_vs","HTTPS","","POST /sites/Home/wservices/JsonWebService.asmx/ReadHighLightsXml HTTP/1.1\r\nAccept: /\r\nContent-Type: application/json; charset=utf-8\r\nReferer: https://www.xxx.lm.sa/sites/Home/\r\nAccept-Language: ar-BR\r\nAccept-Encoding: gzip, deflate\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko\r\nHost: www.xxx.lm.sa\r\nContent-Length: 16\r\nConnection: Keep-Alive\r\nCache-Control: no-cache\r\nCookie: _ga=GA1.3.1270259754.1482313131; ASP.NET_SessionId=idwovi550sj00445isqmdi2f; lang=ar-BR; BIGipServerPool_dp_2014_80=2606606528.20480.0000; TS0145a324=0184b023f4e6a1b6cd88dd0017c4c93b24b52ad5fb24e435fe7e31a47f621ab3e7d8cd0108e60d4b4d276a1b3466c52b8143af3edf200722b6836db714f49e967fdf9dbcdaa8315187828a971fd19905

BR
ML

[magnusbaeck (Magnus Bäck)]

It's basically a standard syslog message (see example in the Logstash documentation) where the message part can be processed by a csv filter.

[mlala (muntazir)]

@magnusbaeck

where the message part can be processed by a csv filter

Hi Magnusbaeck,

actually i already have found a similar article related to my issue but i am still not able to achieve my required output. unfortunately that ticket has been closed. you can search for "Logstash grok customized filter f5 big-ip"

Regards
ML

[magnusbaeck (Magnus Bäck)]

actually i already have found a similar article related to my issue but i am still not able to achieve my required output.

What does your configuration look like? What output do you get?

unfortunately that ticket has been closed. you can search for "Logstash grok customized filter f5 big-ip"

I don't see how that would be helpful.

[mlala (muntazir)]

What does your configuration look like? What output do you get?

Configuration:

input {
udp {
port => 514
type => syslog
}
tcp {
port => 514
type => syslog
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "f5-%{+YYYY.MM.dd}"
manage_template => false
}
}

Output is:

BR
ML

[magnusbaeck (Magnus Bäck)]

%{DATA:syslog_program} matches too much, and looking at the syslog message at least these messages don't have a program and pid at all. Try this instead:

^%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} ASM: %{GREEDYDATA:syslog_message}

Then use a csv filter to parse syslog_message.