Logstash filter verifier - any working example?

terramexx · February 16, 2016, 12:09pm

Continuing the discussion from Logstash test with rspec:

The tool https://github.com/magnusbaeck/logstash-filter-verifier seems to be exactly what I was looking for:

easy install (in compare to rspec)
based on logstash library
simple definition of expected output to provided input

But unfortunately I am unable to run any test with success and always get this error:

2016/02/16 12:58:29 Waiting for child with pid 58724 to terminate.
Error running Logstash: exit status 1. The process gave no output.

Do I need to provide the input somehow next to the argument? Or am I overlooking something? sample example I am trying to run:

logstash-filter-verifier test.json logstash.conf

test.json

{
  "codec": "json",
  "ignore": ["host", "@timestamp"],
  "input": [
    "{\"message\": \"This is a test message\"}"
  ],
  "expected": [
    {
      "message": "This is a test message"
    }
  ]
}

logstash.conf

input {
	stdin {
		codec => "json"
	}
}

filter {
}

output {
	stdout {
		codec => "json"
	}
}

magnusbaeck · February 16, 2016, 12:34pm

Please use Logstash Filter Verifier 1.0.1 which gives better error output when Logstash fails. It also fixes an incompatibility issue with Logstash 2.2.0 (which has also been fixed in Logstash 2.2.1).

Secondly, don't include any inputs or outputs in the configuration file(s) you feed LFV with. Just filters. The documentation should be more clear about this.

terramexx · February 16, 2016, 12:41pm

Hi Magnus,

ok, removed the input and output from my logstash.conf, I am on 2.2.1, but still the same

lukas@apulanta ~/P/k/l/logstash-filter-verifier_1.0.1_linux_amd64> ./logstash-filter-verifier --version
logstash-filter-verifier 1.0.1
lukas@apulanta ~/P/k/l/logstash-filter-verifier_1.0.1_linux_amd64>
./logstash-filter-verifier --logstash-path=logstash test.json logstash.conf --loglevel=DEBUG
2016/02/16 13:39:44 Reading test case file: test.json (/Users/lukas/Projects/kb/logman/logstash-filter-verifier_1.0.1_linux_amd64/test.json)
2016/02/16 13:39:44 Starting "/usr/local/bin/logstash" with args ["logstash" "-w" "1" "--debug" "-e" "input { stdin { codec => \"json\" add_field => {  } } } output { file { path => \"/var/folders/ny/rydql52n7qs32bh56v3dvdz00000gn/T/619215803\" codec => \"json_lines\" } }" "--log" "/var/folders/ny/rydql52n7qs32bh56v3dvdz00000gn/T/910667230" "--config" "logstash.conf"].
2016/02/16 13:39:44 Waiting for child with pid 59259 to terminate.
Error running Logstash: exit status 1. The process gave no output.

magnusbaeck · February 16, 2016, 12:54pm

Wait, what operating system is this? The path to your home directory indicates that it's Mac OS X, but you've downloaded a Linux release of LFV.

terramexx · February 16, 2016, 2:15pm

yeah, that's true, it's mac, and I confirm my mistake too

the story is that I earlier downloaded the amd64 package and then realized my mistake so I changed just the binary with the one from darwin distribution

magnusbaeck · February 16, 2016, 2:37pm

So... you were actually running the Darwin version of the file when you got this error?

terramexx · February 16, 2016, 2:59pm

yeps darwin binary, and before that I was working directly with fresh local make output of the github project

basically when I run logstash with the args directly from the command line the logstash runs and outputs data to some var folder file.

{"message":"test","tags":["_jsonparsefailure"],"@version":"1","@timestamp":"2016-02-16T14:57:28.985Z","host":"apulanta.local"}
{"message":"test","tags":["_jsonparsefailure"],"@version":"1","@timestamp":"2016-02-16T14:57:29.928Z","host":"apulanta.local"}

Any reason for the tags _jsonparsefailure?

I am so excited to start using the tool so hopefully we are close the solution

magnusbaeck · February 16, 2016, 3:22pm

When the message field contains "test" a _jsonparsefailure tag is to be expected since the input string clearly isn't JSON. I can't reproduce what you describe on Linux:

$ cat test.config 
filter { }
$ cat test.json 
{
  "codec": "json",
  "ignore": ["host", "@timestamp"],
  "input": [
    "{\"message\": \"This is a test message\"}"
  ],
  "expected": [
    {
      "message": "This is a test message"
    }
  ]
}
$ logstash-filter-verifier --loglevel=DEBUG test.json test.config                         
2016/02/16 16:20:35 Reading test case file: test.json (/tmp/trash.s2LA/test.json)
2016/02/16 16:20:35 Starting "/opt/logstash/bin/logstash" with args ["/opt/logstash/bin/logstash" "-w" "1" "--debug" "-e" "input { stdin { codec => \"json\" add_field => {  } } } output { file { path => \"/tmp/736256972\" codec => \"json_lines\" } }" "--log" "/tmp/145950395" "--config" "test.config"].
2016/02/16 16:20:35 Waiting for child with pid 16737 to terminate.
Comparing message 1 of test.json...
2016/02/16 16:20:39 Starting "/usr/bin/diff" with args ["-u" "/tmp/399511262/test.json/0/expected" "/tmp/399511262/test.json/0/actual"].

I do have a Mac OS X machine at my disposal so I'll have to try it out there.

terramexx · February 17, 2016, 8:28am

I have just tried on Linux machine, and the same result "The process gave no output.".

magnusbaeck · February 17, 2016, 8:32am

Did you try the exact commands I posted above?

terramexx · February 17, 2016, 3:47pm

yeps, everything as written

few minutes ago we tried new vagrant machine with fresh unix, logstash 2.2.1 install and the logstash-filter-verifier binary worked there as expected

thus there must be some differences which cause the no output error on those mac and unix machines I am using, I will try to hunt it

magnusbaeck · February 17, 2016, 3:49pm

Thanks! I successfully ran this config on one machine yesterday but haven't had time to debug it further.

zdenek.gallistl · February 18, 2016, 9:48am

Hi, I discussed with terramexx and found solution. In file /logstash-filter-verifier/logstash/process.go on line 82 is set one env variable for logstash process. All real evn variables are then lost including JAVA_HOME. Logstahs than not able to find java. I changed this code

c.Env = []string{"TZ=UTC"}

by this:

c.Env = os.Environ()
c.Env = append(c.Env, "TZ=UTC")

Also need to import "io" lib

magnusbaeck · February 18, 2016, 9:53am

Oh, good catch! Thanks!

It was actually intentional to drop the current environment and only set TZ (so that tests are hermetic), but JAVA_HOME should in that case also be propagated.

LFV obviously needs to care about the stdout/stderr of the Logstash program and not just its log.

terramexx · February 18, 2016, 11:02am

hooray, finally we can progress with test automation for logstash filters

big thanks to both of you!

magnusbaeck · February 22, 2016, 9:03pm

Okay, so what I'm planning to do is:

Continue with a clean environment that only sets TZ=UTC but
add a --keep-env option that allows callers to specify the name of additional environment variables that should be propagated to the Logstash processes.

Thereby you can run the tests like this:

logstash-filter-verifier --keep-env JAVA_HOME test.json test.config

If you really want another TZ value you could override the default in the same manner.

I'll release 1.1.0 with this change later this week.

magnusbaeck · February 27, 2016, 7:38pm

Logstash Filter Verifier 1.1.0 with the aforementioned change is now available: https://github.com/magnusbaeck/logstash-filter-verifier/releases/tag/1.1.0

csobchuk · February 24, 2017, 9:46pm

Presently getting the following failure when running make on Ubuntu 14.04 with build essentials, gdb, and golang installed:
>> Setting github.com/alecthomas/gometalinter to version ca13be809964
>> Setting github.com/alecthomas/kingpin to version v2.2.2
>> Setting github.com/mattn/go-shellwords to version f4e566c536cf
>> Setting gopkg.in/matm/v1/gocov-html to version v1.0
>> Setting github.com/mitchellh/packer to version v0.9.0
>> Setting github.com/go-playground/overalls to version 845469c90499
>> Setting github.com/op/go-logging to version dfaf3dff9b63
>> Setting github.com/axw/gocov/gocov to version ac431cdb392e
go build -o logstash-filter-verifier
# github.com/magnusbaeck/logstash-filter-verifier/logstash
logstash/parallel_process.go:57: ts.senderListener.SetUnlinkOnClose undefined (type *net.UnixListener has no field or method SetUnlinkOnClose)
make: *** [logstash-filter-verifier] Error 2
any ideas on what I have to do to correct it?

TIA,

magnusbaeck · February 25, 2017, 10:18am

You need Go 1.8, released last week. I'm going to add a note about this requirement to the readme file but haven't found the time. Sorry for the trouble.

csobchuk · February 27, 2017, 10:41pm

Thanks Magnus.
It compiled once I installed 1.8 and removed the Ubuntu repository version of 1.2.1.

However, now I get an error that logstash wasn't found. Using the example on your Github page for the syslog.json and test.config I ran :
./logstash-filter-verifier syslog.json test.config
and received the following output.
Running tests in syslog.json...
fork/exec /opt/logstash/bin/logstash: no such file or directory
Logstash 5.2 for Ubuntu 14.04, at least, is in /usr/share/logstash on Linux. I suspect I have to change this in the .go file and then recompile? It seems there is no way to discover this from the environment?

Am I on the right track to fix this? I did download it for windows as well - which failed for the same reason.

Topic		Replies	Views
How to use multiple filter files in logstash-filter-verifier command Logstash	1	357	May 19, 2021
Not getting any ouput Logstash	12	1088	July 6, 2017
Are input and output plugins required Logstash	6	433	October 4, 2017
Output of `logstash -e 'input { stdin { } } output { stdout {} }'` Logstash	3	2481	December 31, 2019
Logstash not show any output (again) Logstash	13	3199	July 6, 2017

Logstash filter verifier - any working example?

Related topics