# Grokparse failure mikrotik **URL:** \ **Category:** Logstash\ **Created:** [November 18, 2017, 2:50pm UTC](https://discuss.elastic.co/t/grokparse-failure-mikrotik/108237 "2017-11-18T14:50:07Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Makra](https://avatars.discourse-cdn.com/v4/letter/m/8491ac/32.png) [@Makra](https://discuss.elastic.co/u/Makra)\ **Post date:** [November 18, 2017, 2:50pm UTC](https://discuss.elastic.co/t/grokparse-failure-mikrotik/108237/1 "2017-11-18T14:50:08Z")
Hi I am trying to grok an event log and it looks like following- has\_log: http dstnat: in:\ out:(none), proto TCP (SYN), 10.0.1.215:45306-\>162.235.200.2:80, len 60 I have tested the grok for the above in [https://grokdebug.herokuapp.com/](https://grokdebug.herokuapp.com/) and it is -\> > in:\<%{DATA:uid}\> %{GREEDYDATA:whatever} proto %{WORD:Protocol} %{GREEDYDATA:whatever} %{IP:src\_ip}:%{INT:src\_port}-\>%{IP:dst\_ip}:%{INT:dst\_port} but when i try to filter it inside logstash filter section, it produces grokparsefailure > filter { > grok { > match =\> {"message", "^in:\<%{DATA:uid}\>" } > } > } > or whatever The JSON for the log is { "\_index": "test-2017.11.18", "\_type": "test", "\_id": "AV\_PUSsOjjzJVAmEk2eV", "\_version": 1, "\_score": null, "\_source": { "FACILITY": "user", "HOST": "182.48.91.26", "PRIORITY": "notice", "type": "test", "tags": [ "tcpjson", "\_grokparsefailure" ], **"MESSAGE": "has\_log: http dstnat: in:\ out:(none), proto TCP (SYN), 10.0.1.154:16611-\>123.228.107.253:80, len 60",** "DATE": "Nov 18 13:28:09", "@timestamp": "2017-11-18T13:28:43.990Z", "HOST\_FROM": "172.X.91.26", "port": 46061, "@version": "1", "host": "127.0.0.1", "TAGS": ".source.s\_mikrotik", "SOURCEIP": "182.48.91.26", "PROGRAM": "firewall,info", "LEGACY\_MSGHDR": "firewall,info " }, ......... ......... ......... Looking for a possible solution. Regards ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 18, 2017, 10:41pm UTC](https://discuss.elastic.co/t/grokparse-failure-mikrotik/108237/2 "2017-11-18T22:41:21Z")
> [@Makra](#): > > has\_log: http dstnat: You don't have that in your pattern that I can see? ---
**Author:** ![Makra](https://avatars.discourse-cdn.com/v4/letter/m/8491ac/32.png) [@Makra](https://discuss.elastic.co/u/Makra)\ **Post date:** [November 19, 2017, 3:53am UTC](https://discuss.elastic.co/t/grokparse-failure-mikrotik/108237/3 "2017-11-19T03:53:46Z")
@warkolm I think the pattern [has\_log: http dstnat] in the beginning does not matter because i donot want to look for this pattern in the message. Also I have modified the grok to include the pattern in the beginning, but that results in grokparse failure as well. > has\_log: prerouting: in:\ out:(none), proto TCP (ACK,FIN), 10.0.2.26:2372-\>192.185.90.145:80, NAT (10.0.2.26:2372-\>123.48.91.26:2372)-\>192.185.90.145:80, len 40 =\> > ^has\_log: %{DATA:scheme}: in:\<%{DATA:uid}\> %{GREEDYDATA:whatever} proto %{WORD:Protocol} %{GREEDYDATA:whatever} %{IP:src\_ip}:%{INT:src\_port}-\>%{IP:dst\_ip}:%{INT:dst\_port > > } The above grok for the log message can be verified in [https://grokdebug.herokuapp.com/](https://grokdebug.herokuapp.com/) but the following filter is not working in logstash > filter { > grok { > match =\> { "message" =\> "^has\_log: %{DATA:U}: in:\<%{DATA:uid}\> %{GREEDYDATA:whatever} proto %{WORD:Protocol} %{GREEDYDATA:whatever} %{IP:src\_ip}:%{INT:src\_port}-\>%{IP:dst\_ip}:%{INT:dst\_port}" } > } > } ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 20, 2017, 6:57am UTC](https://discuss.elastic.co/t/grokparse-failure-mikrotik/108237/4 "2017-11-20T06:57:17Z")
Build your expression gradually. Start with `^has_log: %{DATA:U}:` and verify that it gives the expected results, then continue building towards the end. Also, you're using too many DATA and GREEDYDATA for your own good. Excessive use of them carries a heavy performance penalty and could give incorrect matches. ---
**Author:** ![Makra](https://avatars.discourse-cdn.com/v4/letter/m/8491ac/32.png) [@Makra](https://discuss.elastic.co/u/Makra)\ **Post date:** [November 22, 2017, 5:16am UTC](https://discuss.elastic.co/t/grokparse-failure-mikrotik/108237/5 "2017-11-22T05:16:27Z")
@magnusbaeck Tried with parsing from beginning and the result is same. I doubt there may be some other issues other than logstash filter/grok. What i am trying: Logs from Devices/routers -\> Syslog-ng (forwards in JSON format)-\> Logstash **Incoming log entry if i start syslog-ng in verbose mode:** > Incoming log entry; line='firewall,info has\_log: https dstnat: in:\ out:(none), proto TCP (SYN), 10.0.2.71:6220-\>123.10.144.21:443, len 52' > Incoming log entry; line='firewall,info has\_log: prerouting: in:\ out:(none), proto TCP (ACK,FIN), 10.0.1.73:33830-\>123.161.144.88:10086, NAT (10.0.1.73:33830-\>202.48.91.26:33830)-\>123.161.144.88:10086, len 52' > ................ > ................ > ................ **Logstash receives the above logs correctly.** **Logstash config** ``` input { tcp { # codec => json_lines { charset => "UTF-8" } **<= Any other charset ?** codec => json_lines port => 9999 tags => ["tcpjson"] type => "mikrotik" } } filter { grok { match => [ "message", "has_log: %{WORD:uid}:" ] } if "_grokparsefailure" in [tags] { drop {} } } ``` **The grok debugger produces following results for the above logs.** > { > "uid": [ > [ > "prerouting" > ] > ] > } Here is the screenshots of message field in kibana ![Screenshot_2](https://us1.discourse-cdn.com/elastic/original/3X/a/c/ac47d7499cd5633a08a888628e585542ff4da284.png) I have not used DATA or GREEDYDATA this time, so as to minimize the chances of errors. Where is the bug ? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 22, 2017, 6:16am UTC](https://discuss.elastic.co/t/grokparse-failure-mikrotik/108237/6 "2017-11-22T06:16:40Z")
The expression ``` has_log: %{WORD:uid}: ``` obviously doesn't match the line ``` has_log: http dstnat: in:... ``` because WORD only matches one word and "has\_log:" is followed by two words before the colon comes. ---
**Author:** ![Makra](https://avatars.discourse-cdn.com/v4/letter/m/8491ac/32.png) [@Makra](https://discuss.elastic.co/u/Makra)\ **Post date:** [November 22, 2017, 6:29am UTC](https://discuss.elastic.co/t/grokparse-failure-mikrotik/108237/7 "2017-11-22T06:29:13Z")
Hi @magnusbaeck Right, But i have tried with DATA(I have single word logs for has\_log part) but that did not worked. Now i have changed few things like-\> I have removed the format JSON part from the destination configuration of syslog-ng and used syslog input in logstash and it did worked with the same grok. **Old syslog-ng configuration** ``` destination d_mikrotik_json { tcp("127.0.0.1" port(9999) template("$(format-json --scope selected_macros --scope nv_pairs)\n")); }; ``` **New syslog-ng configuration** ``` destination d_mikrotik_json { tcp("127.0.0.1" port(9999)); }; ``` **Old logstash input** ``` input { tcp { # codec => json_lines { charset => "UTF-8" } codec => json_lines port => 9999 tags => ["tcpjson"] type => "mikrotik" } } ``` **Modified Logstash input** ``` input { syslog { host => "127.0.0.1" port => 9999 } } ``` Is that an issue with JSON format/charset ? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 20, 2017, 6:29am UTC](https://discuss.elastic.co/t/grokparse-failure-mikrotik/108237/8 "2017-12-20T06:29:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.