Grokparse failure mikrotik

[Makra (Makara)]

Hi
I am trying to grok an event log and it looks like following-

has_log: http dstnat: in:<pppoe-060_jack> out:(none), proto TCP (SYN), 10.0.1.215:45306->162.235.200.2:80, len 60

I have tested the grok for the above in https://grokdebug.herokuapp.com/
and it is ->

in:<%{DATA:uid}> %{GREEDYDATA:whatever} proto %{WORD:Protocol} %{GREEDYDATA:whatever} %{IP:src_ip}:%{INT:src_port}->%{IP:dst_ip}:%{INT:dst_port}

but when i try to filter it inside logstash filter section, it produces grokparsefailure

filter {
grok {
match => {"message", "^in:<%{DATA:uid}>" }
}
}
or whatever

The JSON for the log is

{
"_index": "test-2017.11.18",
"_type": "test",
"_id": "AV_PUSsOjjzJVAmEk2eV",
"_version": 1,
"_score": null,
"_source": {
"FACILITY": "user",
"HOST": "182.48.91.26",
"PRIORITY": "notice",
"type": "test",
"tags": [
"tcpjson",
"_grokparsefailure"
],
"MESSAGE": "has_log: http dstnat: in:<pppoe-028_jack> out:(none), proto TCP (SYN), 10.0.1.154:16611->123.228.107.253:80, len 60",
"DATE": "Nov 18 13:28:09",
"@timestamp": "2017-11-18T13:28:43.990Z",
"HOST_FROM": "172.X.91.26",
"port": 46061,
"@version": "1",
"host": "127.0.0.1",
"TAGS": ".source.s_mikrotik",
"SOURCEIP": "182.48.91.26",
"PROGRAM": "firewall,info",
"LEGACY_MSGHDR": "firewall,info "
},
.........
.........
.........
Looking for a possible solution.

Regards

[warkolm (Mark Walkom)]

@Makra

has_log: http dstnat:

You don't have that in your pattern that I can see?

[Makra (Makara)]

@warkolm

I think the pattern [has_log: http dstnat] in the beginning does not matter because i donot want to look for this pattern in the message.

Also I have modified the grok to include the pattern in the beginning, but that results in grokparse failure as well.

has_log: prerouting: in:<pppoe-037_jack> out:(none), proto TCP (ACK,FIN), 10.0.2.26:2372->192.185.90.145:80, NAT (10.0.2.26:2372->123.48.91.26:2372)->192.185.90.145:80, len 40

=>

^has_log: %{DATA:scheme}: in:<%{DATA:uid}> %{GREEDYDATA:whatever} proto %{WORD:Protocol} %{GREEDYDATA:whatever} %{IP:src_ip}:%{INT:src_port}->%{IP:dst_ip}:%{INT:dst_port

}

The above grok for the log message can be verified in https://grokdebug.herokuapp.com/

but

the following filter is not working in logstash

filter {
grok {
match => { "message" => "^has_log: %{DATA:U}: in:<%{DATA:uid}> %{GREEDYDATA:whatever} proto %{WORD:Protocol} %{GREEDYDATA:whatever} %{IP:src_ip}:%{INT:src_port}->%{IP:dst_ip}:%{INT:dst_port}" }
}
}

[magnusbaeck (Magnus Bäck)]

Build your expression gradually. Start with ^has_log: %{DATA:U}: and verify that it gives the expected results, then continue building towards the end.

Also, you're using too many DATA and GREEDYDATA for your own good. Excessive use of them carries a heavy performance penalty and could give incorrect matches.

[Makra (Makara)]

@magnusbaeck

Tried with parsing from beginning and the result is same. I doubt there may be some other issues other than logstash filter/grok.

What i am trying:
Logs from Devices/routers -> Syslog-ng (forwards in JSON format)-> Logstash

Incoming log entry if i start syslog-ng in verbose mode:

Incoming log entry; line='firewall,info has_log: https dstnat: in:<pppoe-033_jack> out:(none), proto TCP (SYN), 10.0.2.71:6220->123.10.144.21:443, len 52'
Incoming log entry; line='firewall,info has_log: prerouting: in:<pppoe-049_martin> out:(none), proto TCP (ACK,FIN), 10.0.1.73:33830->123.161.144.88:10086, NAT (10.0.1.73:33830->202.48.91.26:33830)->123.161.144.88:10086, len 52'
................
................
................

Logstash receives the above logs correctly.

Logstash config

input {
        tcp {
              # codec => json_lines { charset => "UTF-8" } **<= Any other charset ?**
              codec => json_lines
              port => 9999
              tags => [ "tcpjson" ]
              type => "mikrotik"
  }
}

filter {
   grok {
              match => [
                                 "message", "has_log: %{WORD:uid}:"
              ]
}
if "_grokparsefailure" in [tags] {
                                                         drop {}
 }
}

The grok debugger produces following results for the above logs.

{
"uid": [
[
"prerouting"
]
]
}

Here is the screenshots of message field in kibana

I have not used DATA or GREEDYDATA this time, so as to minimize the chances of errors.
Where is the bug ?

[magnusbaeck (Magnus Bäck)]

The expression

has_log: %{WORD:uid}:

obviously doesn't match the line

has_log: http dstnat: in:...

because WORD only matches one word and "has_log:" is followed by two words before the colon comes.

[Makra (Makara)]

Hi

@magnusbaeck

Right, But i have tried with DATA(I have single word logs for has_log part) but that did not worked.

Now i have changed few things like->
I have removed the format JSON part from the destination configuration of syslog-ng and used syslog input in logstash and it did worked with the same grok.

Old syslog-ng configuration

destination d_mikrotik_json {
        tcp("127.0.0.1" port(9999) template("$(format-json --scope selected_macros --scope nv_pairs)\n"));
};

New syslog-ng configuration

destination d_mikrotik_json {
         tcp("127.0.0.1" port(9999));
};

Old logstash input

input {
        tcp {
              # codec => json_lines { charset => "UTF-8" }
              codec => json_lines
              port => 9999
              tags => [ "tcpjson" ]
              type => "mikrotik"
  }
}

Modified Logstash input

input {
       syslog {
                host => "127.0.0.1"
                port => 9999
       }
}

Is that an issue with JSON format/charset ?