# Logstash filter debugging workflow **URL:** \ **Category:** Logstash\ **Created:** [November 22, 2017, 1:58am UTC](https://discuss.elastic.co/t/logstash-filter-debugging-workflow/108632 "2017-11-22T01:58:22Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![ajhstn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajhstn/32/24629_2.png) [@ajhstn](https://discuss.elastic.co/u/ajhstn)\ **Post date:** [November 22, 2017, 1:58am UTC](https://discuss.elastic.co/t/logstash-filter-debugging-workflow/108632/1 "2017-11-22T01:58:22Z")
hello, how does one test {filters} quickly making small iterative changes to the .conf file and seeing results? Eg my current workflow is open the .conf file, make a change, save it, restart logstash, wait for a matching event, stop logstash, make more changes, restart logstash....repeat....repeat.. Is there a better way? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 22, 2017, 6:25am UTC](https://discuss.elastic.co/t/logstash-filter-debugging-workflow/108632/2 "2017-11-22T06:25:33Z")
I suggest use of [Logstash Filter Verifier](https://github.com/magnusbaeck/logstash-filter-verifier). (Caveat: I'm the author.) ---
**Author:** ![ajhstn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajhstn/32/24629_2.png) [@ajhstn](https://discuss.elastic.co/u/ajhstn)\ **Post date:** [November 22, 2017, 6:50am UTC](https://discuss.elastic.co/t/logstash-filter-debugging-workflow/108632/3 "2017-11-22T06:50:11Z")
@magnusbaeck strikes again!! Thanks for that, awesome! ---
**Author:** ![ajhstn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajhstn/32/24629_2.png) [@ajhstn](https://discuss.elastic.co/u/ajhstn)\ **Post date:** [November 22, 2017, 10:34pm UTC](https://discuss.elastic.co/t/logstash-filter-debugging-workflow/108632/4 "2017-11-22T22:34:35Z")
Hi Magnus, i have tried to use logstash verifier but am doing something wrong, can you please help? my working dir ```auto 23/11/2017 09:07 AM 82 filter.conf 22/05/2017 06:23 AM 10,173 LICENSE 22/05/2017 06:23 AM 5,341,184 logstash-filter-verifier.exe 22/05/2017 06:23 AM 12,745 README.md 23/11/2017 09:20 AM 485 syslog-auth.json ``` my testcase file ```auto { "fields": { "type": "syslog" }, "testcases": [ { "input": "Oct 6 20:55:29 myhost myprogram[31993]: This is a test message", "expected": [ { "@timestamp": "2015-10-06T20:55:29.000Z", "host": "myhost", "message": "This is a test message", "pid": 31993, "program": "myprogram", "type": "syslog" } ] } ] } ``` my filter file ```auto filter { grok { match => {"message" => "%{SYSLOGBASE}" } } } ``` my command line to run the tool ```auto logstash-filter-verifier.exe ./syslog-auth.json filter.conf --logstash-path="c:\tmp\logstash-5.5.0\bin\logstash" ``` I initially run the tool using the `input` and `expected` fields, but get this error. ```auto Running tests in syslog-auth.json... Testcase failed, continuing with the rest: Expected 1 event(s), got 0 instead. one or more testcases failed ``` So i reworked the `syslog-auth.json` testcase file to include the `testcases` array, but now get this new error. ```auto Error reading/unmarshalling ./syslog-auth.json: json: cannot unmarshal string into Go struct field TestCase.input of type []string ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 23, 2017, 6:35pm UTC](https://discuss.elastic.co/t/logstash-filter-debugging-workflow/108632/5 "2017-11-23T18:35:42Z")
The `input` key should points to an array of strings, so the testcase file should look like this: ```auto { "fields": { "type": "syslog" }, "testcases": [ { "input": [ "Oct 6 20:55:29 myhost myprogram[31993]: This is a test message" ], "expected": [ { "@timestamp": "2015-10-06T20:55:29.000Z", "host": "myhost", "message": "This is a test message", "pid": 31993, "program": "myprogram", "type": "syslog" } ] } ] } ``` ---
**Author:** ![ajhstn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajhstn/32/24629_2.png) [@ajhstn](https://discuss.elastic.co/u/ajhstn)\ **Post date:** [November 24, 2017, 1:28am UTC](https://discuss.elastic.co/t/logstash-filter-debugging-workflow/108632/6 "2017-11-24T01:28:35Z")
Thanks for that, i have updated as you mentioned. Now i am still getting ```auto Running tests in syslog-auth.json... Testcase failed, continuing with the rest: Expected 1 event(s), got 0 instead. one or more testcases failed ``` ---
**Author:** ![ajhstn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajhstn/32/24629_2.png) [@ajhstn](https://discuss.elastic.co/u/ajhstn)\ **Post date:** [November 24, 2017, 6:15am UTC](https://discuss.elastic.co/t/logstash-filter-debugging-workflow/108632/7 "2017-11-24T06:15:34Z")
Hi Magnus, its unclear to me in the documentation, but does logstash-verifier depend on logstash binary? If so, then i i assume i either have to stop logstash from running to use it in the verifier correct? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 24, 2017, 6:32am UTC](https://discuss.elastic.co/t/logstash-filter-debugging-workflow/108632/8 "2017-11-24T06:32:14Z")
Not sure what's going on in your case. The files you posted (with the adjustment I made) worked for me with LFV v1.3.0 and Logstash 5.5.1 on Linux. Perhaps there's a Windows compatibility problem? I have access to a Windows machine that I can try it on this weekend. > its unclear to me in the documentation, but does logstash-verifier depend on logstash binary? It does. > If so, then i i assume i either have to stop logstash from running to use it in the verifier correct? No, they're supposed to run independently. However, depending on how you've installed Logstash 5 there might be a problem where they share the same data directory (which would prevent Logstash from starting when invoked from LFV). I have a number of patches queued up (and a few more to be written) that makes sure that Logstash is run completely sandboxed when started from LFV. I really hope I can get to completing that patch series this weekend. As a workaround until I've released 1.4.0 you can unpack the Logstash distribution in a separate directory that you use only for LFV. That _should_ fix things. ---
**Author:** ![ajhstn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajhstn/32/24629_2.png) [@ajhstn](https://discuss.elastic.co/u/ajhstn)\ **Post date:** [November 24, 2017, 6:35am UTC](https://discuss.elastic.co/t/logstash-filter-debugging-workflow/108632/9 "2017-11-24T06:35:14Z")
Thank you i will try that, much appreciated, and excellent support 🙂 ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 27, 2017, 6:31am UTC](https://discuss.elastic.co/t/logstash-filter-debugging-workflow/108632/10 "2017-11-27T06:31:43Z")
Did you get things going with the workaround? If not, please run LFV with `--loglevel DEBUG` and post the results. ---
**Author:** ![ajhstn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ajhstn/32/24629_2.png) [@ajhstn](https://discuss.elastic.co/u/ajhstn)\ **Post date:** [November 27, 2017, 8:02pm UTC](https://discuss.elastic.co/t/logstash-filter-debugging-workflow/108632/11 "2017-11-27T20:02:47Z")
Hi @magnusbaeck thank you for the nudge to keep at it! I think we are getting somewhere now. I gave up on my local windows machine, and now testing on the production logstash server, using a a fresh unpacked logstash binary for testing. ```auto [root@server scripts]# ./logstash-filter-verifier syslog-auth.json filter.conf --logstash-path=../logstash-5.5.0/bin/l ogstash --loglevel DEBUG 2017/11/28 06:56:36 Reading test case file: syslog-auth.json (/root/scripts/syslog-auth.json) Running tests in syslog-auth.json... 2017/11/28 06:56:36 Prepared configuration file directory /tmp/999381979 with these files: [filter.conf] 2017/11/28 06:56:36 Starting "../logstash-5.5.0/bin/logstash" with args ["-w" "1" "--debug" "-e" "input { stdin { codec => \"line\" add_field => { \"type\" => \"syslog\" } } } output { file { path => \"/tmp/302415873\" codec => \"json_lines\" } }" "-f" "/tmp/999381979" "-l" "/tmp/411917676"]. 2017/11/28 06:56:36 Waiting for child with pid 64730 to terminate. Comparing message 1 of 1 from syslog-auth.json... 2017/11/28 06:56:51 Starting "/usr/bin/diff" with args ["-u" "/tmp/842718078/syslog-auth.json/1/expected" "/tmp/842718078/syslog-auth.json/1/actual"]. --- /tmp/842718078/syslog-auth.json/1/expected 2017-11-28 06:56:51.828224492 +1100 +++ /tmp/842718078/syslog-auth.json/1/actual 2017-11-28 06:56:51.828224492 +1100 @@ -1,8 +1,10 @@ { - "@timestamp": "2015-10-06T20:55:29.000Z", - "host": "myhost", - "message": "This is a test message", - "pid": 31993, + "@timestamp": "2017-11-27T19:56:48.758Z", + "host": "server.domain.net", + "logsource": "myhost", + "message": "Oct 6 20:55:29 myhost myprogram[31993]: This is a test message", + "pid": "31993", "program": "myprogram", + "timestamp": "Oct 6 20:55:29", "type": "syslog" } 2017/11/28 06:56:51 Child with pid 64791 failed: exit status 1 Testcase failed, continuing with the rest: 1 message(s) did not match the expectations. one or more testcases failed [root@servber scripts]# ``` I **think** from here ill hack away at the filter as its all on me now 🙂 Thank you! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 25, 2017, 8:03pm UTC](https://discuss.elastic.co/t/logstash-filter-debugging-workflow/108632/12 "2017-12-25T20:03:29Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.