Ruby filter plugin: how to indicate failure

Elastic Stack Logstash
1

Hi,
currently I use a small Ruby filter script to request a REST API and get some data.
If valid data is returned the script is "successful" and the ruby filter applies the add_tag option and adds some tags. But of course it is possible that no data is found and the script is not successful and thus the filter should not apply any tags.
How to indicate inside the ruby script whether the filter is successful or not?
Currently I raise an exception for invalid or no data, but that causes an error log for each affected event and the _rubyexception tag is added which both is undesirable.
Can you please help me out with that?
Thank you!

2

The upcoming release of Logstash 6.6.0 will include the new http filter (docs). You can install this on older versions of Logstash though.
bin/logstash-plugin install logstash-filter-http

You might like to try it as a solution instead.

If the http filter does not work for you then read on.
This is what the ruby filter does internally.

  def inline_script(event, &block)
    filter_method(event, &block)
    filter_matched(event)
  rescue Exception => e
    @logger.error("Ruby exception occurred: #{e}")
    event.tag(@tag_on_exception)
  end

filter_method is a generated method encapsulating your ruby filter code.
filter_matched is where the add_field, add_tag, remove_field and remove_tag functions are executed.

You have three scenarios.

  1. All good
  2. HTTP call worked but invalid data is returned
  3. Something goes wrong and an exception is raised in filter_method

For 1) and 2) you need to handle the add_tag yourself in the code block when you have invalid data.
event.tag('invalid_data_http')
For 3) if you don't want an error message logged this you should wrap your code in the same begin rescue end construct.

begin
    # your http code here
  rescue Exception => e
    event.tag('http_rest_call_failed')
  end

If you do this make sure you remove the standard add_field, add_tag, remove_field and remove_tag settings from the ruby filter settings.

3

Okay, I see, thanks. Guess I'll wait for the filter plugin. When is the release (planned)?

4

You can install this on older versions of Logstash though.
bin/logstash-plugin install logstash-filter-http

The plugin has been uploaded. You can use it now. When we say released, we mean - it is the first time the plugin is included in the Logstash distribution by default and it is mentioned in the release notes.

The plugin is in the download location and the docs are in Elastic's docs on Logstash plugins.

5

Ay, great.

6

This plugin works very well and it's damn fast. Thanks!

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.