Logstash filter debugging workflow

hello, how does one test {filters} quickly making small iterative changes to the .conf file and seeing results? Eg my current workflow is open the .conf file, make a change, save it, restart logstash, wait for a matching event, stop logstash, make more changes, restart logstash....repeat....repeat.. Is there a better way?

I suggest use of Logstash Filter Verifier. (Caveat: I'm the author.)

1 Like

@magnusbaeck strikes again!! Thanks for that, awesome!

Hi Magnus,

i have tried to use logstash verifier but am doing something wrong, can you please help?

my working dir

23/11/2017  09:07 AM                82 filter.conf
22/05/2017  06:23 AM            10,173 LICENSE
22/05/2017  06:23 AM         5,341,184 logstash-filter-verifier.exe
22/05/2017  06:23 AM            12,745 README.md
23/11/2017  09:20 AM               485 syslog-auth.json

my testcase file

{
  "fields": {
    "type": "syslog"
  },
  "testcases": [
      {
          "input": "Oct  6 20:55:29 myhost myprogram[31993]: This is a test message",
          "expected": [
            {
              "@timestamp": "2015-10-06T20:55:29.000Z",
              "host": "myhost",
              "message": "This is a test message",
              "pid": 31993,
              "program": "myprogram",
              "type": "syslog"
            }
          ]
      }
  ]
}

my filter file

 filter {
    grok {
        match => {"message" => "%{SYSLOGBASE}" }
    }
 }

my command line to run the tool

logstash-filter-verifier.exe ./syslog-auth.json filter.conf --logstash-path="c:\tmp\logstash-5.5.0\bin\logstash"

I initially run the tool using the input and expected fields, but get this error.

Running tests in syslog-auth.json...
Testcase failed, continuing with the rest: Expected 1 event(s), got 0 instead.
one or more testcases failed

So i reworked the syslog-auth.json testcase file to include the testcases array, but now get this new error.

Error reading/unmarshalling ./syslog-auth.json: json: cannot unmarshal string into Go struct field TestCase.input of type []string

The input key should points to an array of strings, so the testcase file should look like this:

{
  "fields": {
    "type": "syslog"
  },
  "testcases": [
      {
          "input": [
            "Oct  6 20:55:29 myhost myprogram[31993]: This is a test message"
          ],
          "expected": [
            {
              "@timestamp": "2015-10-06T20:55:29.000Z",
              "host": "myhost",
              "message": "This is a test message",
              "pid": 31993,
              "program": "myprogram",
              "type": "syslog"
            }
          ]
      }
  ]
}

Thanks for that, i have updated as you mentioned. Now i am still getting

Running tests in syslog-auth.json...
Testcase failed, continuing with the rest: Expected 1 event(s), got 0 instead.
one or more testcases failed

Hi Magnus, its unclear to me in the documentation, but does logstash-verifier depend on logstash binary?

If so, then i i assume i either have to stop logstash from running to use it in the verifier correct?

Not sure what's going on in your case. The files you posted (with the adjustment I made) worked for me with LFV v1.3.0 and Logstash 5.5.1 on Linux. Perhaps there's a Windows compatibility problem? I have access to a Windows machine that I can try it on this weekend.

its unclear to me in the documentation, but does logstash-verifier depend on logstash binary?

It does.

If so, then i i assume i either have to stop logstash from running to use it in the verifier correct?

No, they're supposed to run independently. However, depending on how you've installed Logstash 5 there might be a problem where they share the same data directory (which would prevent Logstash from starting when invoked from LFV). I have a number of patches queued up (and a few more to be written) that makes sure that Logstash is run completely sandboxed when started from LFV. I really hope I can get to completing that patch series this weekend.

As a workaround until I've released 1.4.0 you can unpack the Logstash distribution in a separate directory that you use only for LFV. That should fix things.

1 Like

Thank you i will try that, much appreciated, and excellent support :slight_smile:

Did you get things going with the workaround? If not, please run LFV with --loglevel DEBUG and post the results.

Hi @magnusbaeck thank you for the nudge to keep at it! I think we are getting somewhere now. I gave up on my local windows machine, and now testing on the production logstash server, using a a fresh unpacked logstash binary for testing.

[root@server scripts]# ./logstash-filter-verifier syslog-auth.json filter.conf --logstash-path=../logstash-5.5.0/bin/l
ogstash  --loglevel DEBUG
2017/11/28 06:56:36 Reading test case file: syslog-auth.json (/root/scripts/syslog-auth.json)
Running tests in syslog-auth.json...
2017/11/28 06:56:36 Prepared configuration file directory /tmp/999381979 with these files: [filter.conf]
2017/11/28 06:56:36 Starting "../logstash-5.5.0/bin/logstash" with args ["-w" "1" "--debug" "-e" "input { stdin { codec => \"line\" add_field => { \"type\" => \"syslog\" } } } output { file { path => \"/tmp/302415873\" codec => \"json_lines\" } }" "-f" "/tmp/999381979" "-l" "/tmp/411917676"].
2017/11/28 06:56:36 Waiting for child with pid 64730 to terminate.
Comparing message 1 of 1 from syslog-auth.json...
2017/11/28 06:56:51 Starting "/usr/bin/diff" with args ["-u" "/tmp/842718078/syslog-auth.json/1/expected" "/tmp/842718078/syslog-auth.json/1/actual"].
--- /tmp/842718078/syslog-auth.json/1/expected  2017-11-28 06:56:51.828224492 +1100
+++ /tmp/842718078/syslog-auth.json/1/actual    2017-11-28 06:56:51.828224492 +1100
@@ -1,8 +1,10 @@
 {
-  "@timestamp": "2015-10-06T20:55:29.000Z",
-  "host": "myhost",
-  "message": "This is a test message",
-  "pid": 31993,
+  "@timestamp": "2017-11-27T19:56:48.758Z",
+  "host": "server.domain.net",
+  "logsource": "myhost",
+  "message": "Oct  6 20:55:29 myhost myprogram[31993]: This is a test message",
+  "pid": "31993",
   "program": "myprogram",
+  "timestamp": "Oct  6 20:55:29",
   "type": "syslog"
 }
2017/11/28 06:56:51 Child with pid 64791 failed: exit status 1
Testcase failed, continuing with the rest: 1 message(s) did not match the expectations.
one or more testcases failed
[root@servber scripts]#

I think from here ill hack away at the filter as its all on me now :slight_smile:

Thank you!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.