Logstash filter plugin patterns-core


(Penny) #1

Hi :grin:

I installed the logstash-plugins/logstash-patterns-core to the /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.1.10/
The pattern I tested it work on the http://grokconstructor.appspot.com/do/constructionstep and http://grokdebug.herokuapp.com/ .
I would like to ask why I can't parse the log using logstash filter plugin patterns-core.
Here is my input, filter and output.

Original Log:

Jun 18 14:50:35 0.0.0.0/0.0.0.0 pfsp: anomaly ICMP_Misuse id 123 status ongoing severity 5 classification high impact "13.90 Mbps/18.19 Kpps" src 0.0.0.0/0 All dst 0.0.0.0/0 abc start 2015-06-17 07:10:07 +0000 duration 179 percent 181.870000 rate 10000 rateUnit pps protocol icmp flags nil url https://abc/page?id=alert_view&alert_id=123, (managed object "NXG_INFRA"), (parent managed object "nil")

input {
          lumberjack {
            port => 5000
            type => "logs"
            ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
            ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
          }
}
filter {
    	if [type] == "attacklog" {
        		grok {
    		patterns_dir => "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-0.1.10/patterns"
          		match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{IP:IPAddress}/%{IP:SourceIP} %{PROG:Program}: %{CISCO_REASON:AttackType}%{QS:AttackSize} %{GREEDYDATA:Message}" }
        		}
                    date {
                            match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
                    }
    	}	
}
output {
      elasticsearch { host => localhost }
      stdout { codec => rubydebug }
}

Stdout log:

{
       "message" => "Jun 18 14:50:35 0.0.0.0/0.0.0.0 pfsp: anomaly ICMP_Misuse id 123 status ongoing severity 5 classification high impact 13.90 Mbps/18.19 Kpps src 0.0.0.0/0 All dst 0.0.0.0/0 abc start 2015-06-17 07:10:07 +0000 duration 179 percent 181.870000 rate 10000 rateUnit pps protocol icmp flags nil url https://abc/page?id=alert_view&alert_id=123, (managed object NXG_INFRA), (parent managed object nil)",
      "@version" => "1",
    "@timestamp" => "2015-06-18T10:32:07.385Z",
          "type" => "attacklog",
          "file" => "/var/log/attack.log",
          "host" => "syslogserver",
        "offset" => "7210",
          "tags" => [
        [0] "_grokparsefailure"
    ]
}

Thanks a lot.


(Penny) #2

Its fixed. Formerly, logstash have installed the plugin logstash-patterns-core.
It should touch the file in the /opt/logstash/patterns, and you can define the your grok pattern to use in the filter.


(system) #3