[ANNOUNCEMENT] logstash-filter-test

[noamraph (Noam Yorav Raphael)]

Hi, I tried logstash-filter-verifier for a few days and grew frustrated with it. So I wrote my own tester: logstash-filter-test. I find it to be much simpler and easier. Here is an example test suite file:

[
  // Test a message with a value
  [
    {
      "message": "<<2018-06-11 13:45:39,127+0300>> TAU = 6.2831853"
    },
    {
      "date": "2018-06-11 13:45:39,127+0300",
      "@timestamp": "2018-06-11T10:45:39.127Z",
      "msg_text": "TAU = 6.2831853",
      "metric": "TAU",
      "value": 6.2831853,
    }
  ],
]

You can also just run the filter from Python:

>>> from logstash_filter_run import logstash_filter_run
>>> logstash_filter_run([{"a": 3}], 'filter { mutate { copy => { "a" => "b" } } }')
[{'@timestamp': '2018-07-10T18:26:58.411Z',
  '@version': '1',
  'a': 3,
  'b': 3,
  'host': 'myhost'}]

I hope you enjoy it!
Noam

[magnusbaeck (Magnus Bäck)]

I tried logstash-filter-verifier for a few days and grew frustrated with it.

How come?

[noamraph (Noam Yorav Raphael)]

A few annoyances that I think of right now:

• It took me some time to understand the "testcase" format (the new format where the output comes immediately after its input). There are no examples, and for some reason each input should be in its own 1-length list.
• I needed a new file for each case of surrounding fields. My filter adds timezone based on host name, and I needed a separate file for each host name.
• Again, I needed a separate file for each set of fields I wanted to test.
• In order not to wait 30 seconds for each file, I had to find a command line argument, and try to convince myself that the caveat that is described in the documentation doesn't apply.
• I had to resort to a very strange hack in order to be able to send a multi-line message. (I used a string that is a json-encoded string.)
• I like being able to add comments and trailing commas to the json, to describe the test cases and to not have to bother with adding and removing commas after adding or removing lines.
• I like being able to just put in a json document and see the result. I can now do it from a python interpreter using logstash_filter_run().

Thanks,
Noam