hello, how does one test {filters} quickly making small iterative changes to the .conf file and seeing results? Eg my current workflow is open the .conf file, make a change, save it, restart logstash, wait for a matching event, stop logstash, make more changes, restart logstash....repeat....repeat.. Is there a better way?
I suggest use of Logstash Filter Verifier. (Caveat: I'm the author.)
@magnusbaeck strikes again!! Thanks for that, awesome!
Hi Magnus,
i have tried to use logstash verifier but am doing something wrong, can you please help?
my working dir
23/11/2017 09:07 AM 82 filter.conf
22/05/2017 06:23 AM 10,173 LICENSE
22/05/2017 06:23 AM 5,341,184 logstash-filter-verifier.exe
22/05/2017 06:23 AM 12,745 README.md
23/11/2017 09:20 AM 485 syslog-auth.json
my testcase file
{
"fields": {
"type": "syslog"
},
"testcases": [
{
"input": "Oct 6 20:55:29 myhost myprogram[31993]: This is a test message",
"expected": [
{
"@timestamp": "2015-10-06T20:55:29.000Z",
"host": "myhost",
"message": "This is a test message",
"pid": 31993,
"program": "myprogram",
"type": "syslog"
}
]
}
]
}
my filter file
filter {
grok {
match => {"message" => "%{SYSLOGBASE}" }
}
}
my command line to run the tool
logstash-filter-verifier.exe ./syslog-auth.json filter.conf --logstash-path="c:\tmp\logstash-5.5.0\bin\logstash"
I initially run the tool using the input and expected fields, but get this error.
Running tests in syslog-auth.json...
Testcase failed, continuing with the rest: Expected 1 event(s), got 0 instead.
one or more testcases failed
So i reworked the syslog-auth.json testcase file to include the testcases array, but now get this new error.
Error reading/unmarshalling ./syslog-auth.json: json: cannot unmarshal string into Go struct field TestCase.input of type []string
The input key should points to an array of strings, so the testcase file should look like this:
{
"fields": {
"type": "syslog"
},
"testcases": [
{
"input": [
"Oct 6 20:55:29 myhost myprogram[31993]: This is a test message"
],
"expected": [
{
"@timestamp": "2015-10-06T20:55:29.000Z",
"host": "myhost",
"message": "This is a test message",
"pid": 31993,
"program": "myprogram",
"type": "syslog"
}
]
}
]
}
Thanks for that, i have updated as you mentioned. Now i am still getting
Running tests in syslog-auth.json...
Testcase failed, continuing with the rest: Expected 1 event(s), got 0 instead.
one or more testcases failed
Hi Magnus, its unclear to me in the documentation, but does logstash-verifier depend on logstash binary?
If so, then i i assume i either have to stop logstash from running to use it in the verifier correct?
Not sure what's going on in your case. The files you posted (with the adjustment I made) worked for me with LFV v1.3.0 and Logstash 5.5.1 on Linux. Perhaps there's a Windows compatibility problem? I have access to a Windows machine that I can try it on this weekend.
its unclear to me in the documentation, but does logstash-verifier depend on logstash binary?
It does.
If so, then i i assume i either have to stop logstash from running to use it in the verifier correct?
No, they're supposed to run independently. However, depending on how you've installed Logstash 5 there might be a problem where they share the same data directory (which would prevent Logstash from starting when invoked from LFV). I have a number of patches queued up (and a few more to be written) that makes sure that Logstash is run completely sandboxed when started from LFV. I really hope I can get to completing that patch series this weekend.
As a workaround until I've released 1.4.0 you can unpack the Logstash distribution in a separate directory that you use only for LFV. That should fix things.
Thank you i will try that, much appreciated, and excellent support 
Did you get things going with the workaround? If not, please run LFV with --loglevel DEBUG and post the results.
Hi @magnusbaeck thank you for the nudge to keep at it! I think we are getting somewhere now. I gave up on my local windows machine, and now testing on the production logstash server, using a a fresh unpacked logstash binary for testing.
[root@server scripts]# ./logstash-filter-verifier syslog-auth.json filter.conf --logstash-path=../logstash-5.5.0/bin/l
ogstash --loglevel DEBUG
2017/11/28 06:56:36 Reading test case file: syslog-auth.json (/root/scripts/syslog-auth.json)
Running tests in syslog-auth.json...
2017/11/28 06:56:36 Prepared configuration file directory /tmp/999381979 with these files: [filter.conf]
2017/11/28 06:56:36 Starting "../logstash-5.5.0/bin/logstash" with args ["-w" "1" "--debug" "-e" "input { stdin { codec => \"line\" add_field => { \"type\" => \"syslog\" } } } output { file { path => \"/tmp/302415873\" codec => \"json_lines\" } }" "-f" "/tmp/999381979" "-l" "/tmp/411917676"].
2017/11/28 06:56:36 Waiting for child with pid 64730 to terminate.
Comparing message 1 of 1 from syslog-auth.json...
2017/11/28 06:56:51 Starting "/usr/bin/diff" with args ["-u" "/tmp/842718078/syslog-auth.json/1/expected" "/tmp/842718078/syslog-auth.json/1/actual"].
--- /tmp/842718078/syslog-auth.json/1/expected 2017-11-28 06:56:51.828224492 +1100
+++ /tmp/842718078/syslog-auth.json/1/actual 2017-11-28 06:56:51.828224492 +1100
@@ -1,8 +1,10 @@
{
- "@timestamp": "2015-10-06T20:55:29.000Z",
- "host": "myhost",
- "message": "This is a test message",
- "pid": 31993,
+ "@timestamp": "2017-11-27T19:56:48.758Z",
+ "host": "server.domain.net",
+ "logsource": "myhost",
+ "message": "Oct 6 20:55:29 myhost myprogram[31993]: This is a test message",
+ "pid": "31993",
"program": "myprogram",
+ "timestamp": "Oct 6 20:55:29",
"type": "syslog"
}
2017/11/28 06:56:51 Child with pid 64791 failed: exit status 1
Testcase failed, continuing with the rest: 1 message(s) did not match the expectations.
one or more testcases failed
[root@servber scripts]#
I think from here ill hack away at the filter as its all on me now
Thank you!