# Logstash-filter-verifier vs grok debugger differences

**URL:** <https://discuss.elastic.co/t/logstash-filter-verifier-vs-grok-debugger-differences/94389>\
**Category:** Logstash\
**Created:** [July 24, 2017, 7:56pm UTC](https://discuss.elastic.co/t/logstash-filter-verifier-vs-grok-debugger-differences/94389 "2017-07-24T19:56:57Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![csobchuk](https://avatars.discourse-cdn.com/v4/letter/c/a8b319/32.png) [@csobchuk](https://discuss.elastic.co/u/csobchuk)\
**Post date:** [July 24, 2017, 7:56pm UTC](https://discuss.elastic.co/t/logstash-filter-verifier-vs-grok-debugger-differences/94389/1 "2017-07-24T19:56:57Z")

</div>

Hi I am using logstash 5.2 and the latest logstash-filter-verifier and am noticing a difference between the output of the logstash-filter-verifier and what I submitted to the grok debugger.

Grok Debugger input:  
`"2017-03-29 15:48:48.872 8698 WARNING oslo.service.loopingcall [req-aa38d6b9-c72a-4fbe-b8eb-3340ca9cf353 - - - - -] Function 'neutron.agent.dhcp.agent.DhcpAgentWithStateReport._report_state' run outlasted interval by 0.01 sec"`

Grok Debugger output:  
`https://pastebin.com/s8jhpF1k`

Grok Debugger filter:  
`"%{TIMESTAMP_ISO8601:[@metadata][timestamp]}%{SPACE}%{NUMBER:pid}%{SPACE}?%{LOGLEVEL:loglevel} \[?\b%{NOTSPACE:module}\b\]?%{SPACE}? (?:\[\-\]|\[(?:(%{NOTSPACE:request_id}|\-) (%{NOTSPACE:user}|\-) (%{NOTSPACE:tenant}|\-) (%{NOTSPACE:domain}|\-) (%{NOTSPACE:user_domain}|\-) (%{NOTSPACE:project_domain}|\-)\]))?%{SPACE}?(?:%{IP:instance}|(?<instance>\h{8}-\h{4}-\h{4}-\h{4}-\h{12}))?%{SPACE}?%{DATA:logmessage}%{SPACE}?(?<msg_data>%{NOTSPACE}=%{GREEDYDATA})?"`

Logstash-filter-verifier input line:  
`"2017-03-29 15:48:48.872 8698 WARNING oslo.service.loopingcall [req-aa38d6b9-c72a-4fbe-b8eb-3340ca9cf353 - - - - -] Function 'neutron.agent.dhcp.agent.DhcpAgentWithStateReport._report_state' run outlasted interval by 0.01 sec",`

Logstash-filter-verifier conf:  
filter {  
grok {  
match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:[@metadata][timestamp]}%{SPACE}%{NUMBER:pid}%{SPACE}?%{LOGLEVEL:loglevel} [?\b%{NOTSPACE:module}\b]?%{SPACE}? (?:[-]|[(?:(%{NOTSPACE:request\_id}|-) (%{NOTSPACE:user}|-) (%{NOTSPACE:tenant}|-) (%{NOTSPACE:domain}|-) (%{NOTSPACE:user\_domain}|-) (%{NOTSPACE:project\_domain}|-)]))?%{SPACE}?(?:%{IP:instance}|(?\h{8}-\h{4}-\h{4}-\h{4}-\h{12}))?%{SPACE}?%{DATA:logmessage}%{SPACE}?(?\<msg\_data\>%{NOTSPACE}=%{GREEDYDATA})?" }  
add\_field =\> { "received\_at" =\> "%{@timestamp}" }  
}  
kv {  
source =\> "msg\_data"  
field\_split =\> " ,"  
}  
date { match =\> ["[@metadata][timestamp]", "YYYY-MM-DD HH:mm:ss.sss", "ISO8601"] }  
if [module] == "iso8601.iso8601" {  
drop {}  
}  
}

Logstash-filter-verifier output:  
Comparing message 2 of 100 from neutron.json...  
--- /tmp/473250745/neutron.json/2/expected 2017-07-24 19:40:50.065109719 +0000  
+++ /tmp/473250745/neutron.json/2/actual 2017-07-24 19:40:50.065109719 +0000  
@@ -1,3 +1,15 @@  
{  
- "host": "cfa"  
+ "@timestamp": "2017-03-29T15:48:48.872Z",  
+ "domain": "-",  
+ "host": "cfa",  
+ "loglevel": "WARNING",  
+ "message": "2017-03-29 15:48:48.872 8698 WARNING oslo.service.loopingcall [req-aa38d6b9-c72a-4fbe-b8eb-3340ca9cf353 - - - - -] Function 'neutron.agent.dhcp.agent.DhcpAgentWithStateReport.\_report\_state' run outlasted interval by 0.01 sec",  
+ "module": "oslo.service.loopingcall",  
+ "pid": "8698",  
+ "project\_domain": "-",  
+ "received\_at": "2017-07-24T19:32:08.853Z",  
+ "request\_id": "req-aa38d6b9-c72a-4fbe-b8eb-3340ca9cf353",  
+ "tenant": "-",  
+ "user": "-",  
+ "user\_domain": "-"  
}  
I was expecting the logstash-filter-verifier to be the same as the grok debugger output - specifically that the logmessage would have been captured as:  
"logmessage": [  
[  
"Function 'neutron.agent.dhcp.agent.DhcpAgentWithStateReport.\_report\_state' run outlasted interval by 0.01 sec"  
]  
],

What am I missing, or should it be the same?

---

<div class="post-metadata">

**Author:** ![csobchuk](https://avatars.discourse-cdn.com/v4/letter/c/a8b319/32.png) [@csobchuk](https://discuss.elastic.co/u/csobchuk)\
**Post date:** [July 24, 2017, 11:04pm UTC](https://discuss.elastic.co/t/logstash-filter-verifier-vs-grok-debugger-differences/94389/2 "2017-07-24T23:04:13Z")

</div>

I see I copied the wrong grok filter for the LFV.conf- it should have had`(?<msg_data>%{NOTSPACE}=%{GREEDYDATA})` in it which matches the grok debugger string.  
I think I found out that the difference is that the grok debugger includes "`$`" as a valid endpoint for `%{DATA}` where as logstash `%{DATA}` filter, which is identified as the same "`.*?`", does not include "`$`" as a valid end point (for the "`?`").  
By reordering the msg\_data string and changing the `%{DATA:logmessage}` to`(?<logmessage>[A-Za-z0-9' .:_-&&[^=]]+))` I think I have a workaround that works.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 21, 2017, 11:04pm UTC](https://discuss.elastic.co/t/logstash-filter-verifier-vs-grok-debugger-differences/94389/3 "2017-08-21T23:04:40Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.