hello, how does one test {filters} quickly making small iterative changes to the .conf file and seeing results? Eg my current workflow is open the .conf file, make a change, save it, restart logstash, wait for a matching event, stop logstash, make more changes, restart logstash....repeat....repeat.. Is there a better way?
I suggest use of Logstash Filter Verifier. (Caveat: I'm the author.)
1 Like
@magnusbaeck strikes again!! Thanks for that, awesome!
Hi Magnus,
i have tried to use logstash verifier but am doing something wrong, can you please help?
my working dir
23/11/2017 09:07 AM 82 filter.conf
22/05/2017 06:23 AM 10,173 LICENSE
22/05/2017 06:23 AM 5,341,184 logstash-filter-verifier.exe
22/05/2017 06:23 AM 12,745 README.md
23/11/2017 09:20 AM 485 syslog-auth.json
my testcase file
{
"fields": {
"type": "syslog"
},
"testcases": [
{
"input": "Oct 6 20:55:29 myhost myprogram[31993]: This is a test message",
"expected": [
{
"@timestamp": "2015-10-06T20:55:29.000Z",
"host": "myhost",
"message": "This is a test message",
"pid": 31993,
"program": "myprogram",
"type": "syslog"
}
]
}
]
}
my filter file
filter {
grok {
match => {"message" => "%{SYSLOGBASE}" }
}
}
my command line to run the tool
logstash-filter-verifier.exe ./syslog-auth.json filter.conf --logstash-path="c:\tmp\logstash-5.5.0\bin\logstash"
I initially run the tool using the input and expected fields, but get this error.
Running tests in syslog-auth.json...
Testcase failed, continuing with the rest: Expected 1 event(s), got 0 instead.
one or more testcases failed
So i reworked the syslog-auth.json testcase file to include the testcases array, but now get this new error.
Error reading/unmarshalling ./syslog-auth.json: json: cannot unmarshal string into Go struct field TestCase.input of type []string
The input key should points to an array of strings, so the testcase file should look like this:
{
"fields": {
"type": "syslog"
},
"testcases": [
{
"input": [
"Oct 6 20:55:29 myhost myprogram[31993]: This is a test message"
],
"expected": [
{
"@timestamp": "2015-10-06T20:55:29.000Z",
"host": "myhost",
"message": "This is a test message",
"pid": 31993,
"program": "myprogram",
"type": "syslog"
}
]
}
]
}
Thanks for that, i have updated as you mentioned. Now i am still getting
Running tests in syslog-auth.json...
Testcase failed, continuing with the rest: Expected 1 event(s), got 0 instead.
one or more testcases failed
Hi Magnus, its unclear to me in the documentation, but does logstash-verifier depend on logstash binary?
If so, then i i assume i either have to stop logstash from running to use it in the verifier correct?
Not sure what's going on in your case. The files you posted (with the adjustment I made) worked for me with LFV v1.3.0 and Logstash 5.5.1 on Linux. Perhaps there's a Windows compatibility problem? I have access to a Windows machine that I can try it on this weekend.
its unclear to me in the documentation, but does logstash-verifier depend on logstash binary?
It does.
If so, then i i assume i either have to stop logstash from running to use it in the verifier correct?
No, they're supposed to run independently. However, depending on how you've installed Logstash 5 there might be a problem where they share the same data directory (which would prevent Logstash from starting when invoked from LFV). I have a number of patches queued up (and a few more to be written) that makes sure that Logstash is run completely sandboxed when started from LFV. I really hope I can get to completing that patch series this weekend.
As a workaround until I've released 1.4.0 you can unpack the Logstash distribution in a separate directory that you use only for LFV. That should fix things.
1 Like
Thank you i will try that, much appreciated, and excellent support 
Did you get things going with the workaround? If not, please run LFV with --loglevel DEBUG and post the results.
Hi @magnusbaeck thank you for the nudge to keep at it! I think we are getting somewhere now. I gave up on my local windows machine, and now testing on the production logstash server, using a a fresh unpacked logstash binary for testing.
[root@server scripts]# ./logstash-filter-verifier syslog-auth.json filter.conf --logstash-path=../logstash-5.5.0/bin/l
ogstash --loglevel DEBUG
2017/11/28 06:56:36 Reading test case file: syslog-auth.json (/root/scripts/syslog-auth.json)
Running tests in syslog-auth.json...
2017/11/28 06:56:36 Prepared configuration file directory /tmp/999381979 with these files: [filter.conf]
2017/11/28 06:56:36 Starting "../logstash-5.5.0/bin/logstash" with args ["-w" "1" "--debug" "-e" "input { stdin { codec => \"line\" add_field => { \"type\" => \"syslog\" } } } output { file { path => \"/tmp/302415873\" codec => \"json_lines\" } }" "-f" "/tmp/999381979" "-l" "/tmp/411917676"].
2017/11/28 06:56:36 Waiting for child with pid 64730 to terminate.
Comparing message 1 of 1 from syslog-auth.json...
2017/11/28 06:56:51 Starting "/usr/bin/diff" with args ["-u" "/tmp/842718078/syslog-auth.json/1/expected" "/tmp/842718078/syslog-auth.json/1/actual"].
--- /tmp/842718078/syslog-auth.json/1/expected 2017-11-28 06:56:51.828224492 +1100
+++ /tmp/842718078/syslog-auth.json/1/actual 2017-11-28 06:56:51.828224492 +1100
@@ -1,8 +1,10 @@
{
- "@timestamp": "2015-10-06T20:55:29.000Z",
- "host": "myhost",
- "message": "This is a test message",
- "pid": 31993,
+ "@timestamp": "2017-11-27T19:56:48.758Z",
+ "host": "server.domain.net",
+ "logsource": "myhost",
+ "message": "Oct 6 20:55:29 myhost myprogram[31993]: This is a test message",
+ "pid": "31993",
"program": "myprogram",
+ "timestamp": "Oct 6 20:55:29",
"type": "syslog"
}
2017/11/28 06:56:51 Child with pid 64791 failed: exit status 1
Testcase failed, continuing with the rest: 1 message(s) did not match the expectations.
one or more testcases failed
[root@servber scripts]#
I think from here ill hack away at the filter as its all on me now
Thank you!
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.