Logstash filter verifier - any working example?

Elastic Stack Logstash
1

Continuing the discussion from Logstash test with rspec:

The tool https://github.com/magnusbaeck/logstash-filter-verifier seems to be exactly what I was looking for:

  • easy install (in compare to rspec)
  • based on logstash library
  • simple definition of expected output to provided input

But unfortunately I am unable to run any test with success and always get this error:

2016/02/16 12:58:29 Waiting for child with pid 58724 to terminate.
Error running Logstash: exit status 1. The process gave no output.

Do I need to provide the input somehow next to the argument? Or am I overlooking something? sample example I am trying to run:

logstash-filter-verifier test.json logstash.conf

test.json

{
  "codec": "json",
  "ignore": ["host", "@timestamp"],
  "input": [
    "{\"message\": \"This is a test message\"}"
  ],
  "expected": [
    {
      "message": "This is a test message"
    }
  ]
}

logstash.conf

input {
	stdin {
		codec => "json"
	}
}

filter {
}

output {
	stdout {
		codec => "json"
	}
}
2

Please use Logstash Filter Verifier 1.0.1 which gives better error output when Logstash fails. It also fixes an incompatibility issue with Logstash 2.2.0 (which has also been fixed in Logstash 2.2.1).

Secondly, don't include any inputs or outputs in the configuration file(s) you feed LFV with. Just filters. The documentation should be more clear about this.

3

Hi Magnus,

ok, removed the input and output from my logstash.conf, I am on 2.2.1, but still the same

lukas@apulanta ~/P/k/l/logstash-filter-verifier_1.0.1_linux_amd64> ./logstash-filter-verifier --version
logstash-filter-verifier 1.0.1
lukas@apulanta ~/P/k/l/logstash-filter-verifier_1.0.1_linux_amd64>
./logstash-filter-verifier --logstash-path=logstash test.json logstash.conf --loglevel=DEBUG
2016/02/16 13:39:44 Reading test case file: test.json (/Users/lukas/Projects/kb/logman/logstash-filter-verifier_1.0.1_linux_amd64/test.json)
2016/02/16 13:39:44 Starting "/usr/local/bin/logstash" with args ["logstash" "-w" "1" "--debug" "-e" "input { stdin { codec => \"json\" add_field => {  } } } output { file { path => \"/var/folders/ny/rydql52n7qs32bh56v3dvdz00000gn/T/619215803\" codec => \"json_lines\" } }" "--log" "/var/folders/ny/rydql52n7qs32bh56v3dvdz00000gn/T/910667230" "--config" "logstash.conf"].
2016/02/16 13:39:44 Waiting for child with pid 59259 to terminate.
Error running Logstash: exit status 1. The process gave no output.
4

Wait, what operating system is this? The path to your home directory indicates that it's Mac OS X, but you've downloaded a Linux release of LFV.

5

yeah, that's true, it's mac, and I confirm my mistake too

the story is that I earlier downloaded the amd64 package and then realized my mistake so I changed just the binary with the one from darwin distribution

6

So... you were actually running the Darwin version of the file when you got this error?

7

yeps darwin binary, and before that I was working directly with fresh local make output of the github project

basically when I run logstash with the args directly from the command line the logstash runs and outputs data to some var folder file.

{"message":"test","tags":["_jsonparsefailure"],"@version":"1","@timestamp":"2016-02-16T14:57:28.985Z","host":"apulanta.local"}
{"message":"test","tags":["_jsonparsefailure"],"@version":"1","@timestamp":"2016-02-16T14:57:29.928Z","host":"apulanta.local"}

Any reason for the tags _jsonparsefailure?

I am so excited to start using the tool so hopefully we are close the solution :slight_smile:

8

When the message field contains "test" a _jsonparsefailure tag is to be expected since the input string clearly isn't JSON. I can't reproduce what you describe on Linux:

$ cat test.config 
filter { }
$ cat test.json 
{
  "codec": "json",
  "ignore": ["host", "@timestamp"],
  "input": [
    "{\"message\": \"This is a test message\"}"
  ],
  "expected": [
    {
      "message": "This is a test message"
    }
  ]
}
$ logstash-filter-verifier --loglevel=DEBUG test.json test.config                         
2016/02/16 16:20:35 Reading test case file: test.json (/tmp/trash.s2LA/test.json)
2016/02/16 16:20:35 Starting "/opt/logstash/bin/logstash" with args ["/opt/logstash/bin/logstash" "-w" "1" "--debug" "-e" "input { stdin { codec => \"json\" add_field => {  } } } output { file { path => \"/tmp/736256972\" codec => \"json_lines\" } }" "--log" "/tmp/145950395" "--config" "test.config"].
2016/02/16 16:20:35 Waiting for child with pid 16737 to terminate.
Comparing message 1 of test.json...
2016/02/16 16:20:39 Starting "/usr/bin/diff" with args ["-u" "/tmp/399511262/test.json/0/expected" "/tmp/399511262/test.json/0/actual"].

I do have a Mac OS X machine at my disposal so I'll have to try it out there.

9

I have just tried on Linux machine, and the same result "The process gave no output.".

10

Did you try the exact commands I posted above?

11

yeps, everything as written

few minutes ago we tried new vagrant machine with fresh unix, logstash 2.2.1 install and the logstash-filter-verifier binary worked there as expected

thus there must be some differences which cause the no output error on those mac and unix machines I am using, I will try to hunt it

12

Thanks! I successfully ran this config on one machine yesterday but haven't had time to debug it further.

13

Hi, I discussed with terramexx and found solution. In file /logstash-filter-verifier/logstash/process.go on line 82 is set one env variable for logstash process. All real evn variables are then lost including JAVA_HOME. Logstahs than not able to find java. I changed this code

c.Env = []string{"TZ=UTC"}

by this:

c.Env = os.Environ()
c.Env = append(c.Env, "TZ=UTC")

Also need to import "io" lib

14

Oh, good catch! Thanks!

It was actually intentional to drop the current environment and only set TZ (so that tests are hermetic), but JAVA_HOME should in that case also be propagated.

LFV obviously needs to care about the stdout/stderr of the Logstash program and not just its log.

15

hooray, finally we can progress with test automation for logstash filters

big thanks to both of you!

16

Okay, so what I'm planning to do is:

  • Continue with a clean environment that only sets TZ=UTC but
  • add a --keep-env option that allows callers to specify the name of additional environment variables that should be propagated to the Logstash processes.

Thereby you can run the tests like this:

logstash-filter-verifier --keep-env JAVA_HOME test.json test.config

If you really want another TZ value you could override the default in the same manner.

I'll release 1.1.0 with this change later this week.

17

Logstash Filter Verifier 1.1.0 with the aforementioned change is now available: https://github.com/magnusbaeck/logstash-filter-verifier/releases/tag/1.1.0

18

Presently getting the following failure when running make on Ubuntu 14.04 with build essentials, gdb, and golang installed:
>> Setting github.com/alecthomas/gometalinter to version ca13be809964
>> Setting github.com/alecthomas/kingpin to version v2.2.2
>> Setting github.com/mattn/go-shellwords to version f4e566c536cf
>> Setting gopkg.in/matm/v1/gocov-html to version v1.0
>> Setting github.com/mitchellh/packer to version v0.9.0
>> Setting github.com/go-playground/overalls to version 845469c90499
>> Setting github.com/op/go-logging to version dfaf3dff9b63
>> Setting github.com/axw/gocov/gocov to version ac431cdb392e
go build -o logstash-filter-verifier
# github.com/magnusbaeck/logstash-filter-verifier/logstash
logstash/parallel_process.go:57: ts.senderListener.SetUnlinkOnClose undefined (type *net.UnixListener has no field or method SetUnlinkOnClose)
make: *** [logstash-filter-verifier] Error 2
any ideas on what I have to do to correct it?

TIA,

19

You need Go 1.8, released last week. I'm going to add a note about this requirement to the readme file but haven't found the time. Sorry for the trouble.

20

Thanks Magnus.
It compiled once I installed 1.8 and removed the Ubuntu repository version of 1.2.1.

However, now I get an error that logstash wasn't found. Using the example on your Github page for the syslog.json and test.config I ran :
./logstash-filter-verifier syslog.json test.config
and received the following output.
Running tests in syslog.json...
fork/exec /opt/logstash/bin/logstash: no such file or directory
Logstash 5.2 for Ubuntu 14.04, at least, is in /usr/share/logstash on Linux. I suspect I have to change this in the .go file and then recompile? It seems there is no way to discover this from the environment?

Am I on the right track to fix this? I did download it for windows as well - which failed for the same reason.