How to stop logstash when ruby filter encountered error

Elastic Stack Logstash
1

Hi,

May i know how to stop logstash from indexing when error found inside ruby filter?

i have following config file which will call external python script to process data and insert the records into elasticsearch.

It has no issue, except that when the ruby filter encounter error, logstash will not stop but continue to index empty records into elasticsearch. How to stop the process?

sample code:

input {
	file {
		path => "path/logfiledata.txt" 
		start_position => "beginning"
		sincedb_path => "NUL"
	}
}
filter {
    if [message] =~ /.+/ {
       ruby {
            code => 'require "open3"
                data = "\"" + event.get("message").strip + "\""
                cmd = "python /path/pythonscritp.py #{data}"
                Open3.popen3(cmd) do |stdin, stdout, stderr, wait_thr|
                  begin
                    event.set("process_result", stdout.readline)
                  rescue StandardError => e
                    logger.error("Exception: " + "class name: " + e.class.name + " | " + "error message: " + stderr.read)
                  end
                end'
       }
     }
    mutate {
        # to remove carrige return, new line, tab, double quote, square/curly brackets
        gsub => ["process_result","[\t\n\r]", ""]
        gsub => ['process_result', '"', "" ]
        gsub => ["process_result", "[\{\}\[\]]", "" ]
    }
    kv {
        source => "process_result"
        field_split => ","
        value_split => ":"
    }
    mutate {
       remove_field => [ "process_result" ]
       remove_field => [ "message" ]
       remove_field => [ "path" ]
       remove_field => [ "@version" ]
       remove_field => [ "host" ]
 }
 
}

output{ 
  elasticsearch {
      hosts => "localhost:9200"
      index => "redemption_reason_%{+YYYY_MM_dd}"
      user => "elastic" 
      password => "changeme"
      ssl => true
      ssl_certificate_verification => true
      cacert => "certificate.crt"
  }
  stdout{}
}
2

you can try to add a field or a tag everytime ruby filter throws error, then on your output set a condition to only send the output to ES if the field or tag doesn’t exist. you can direct the failed event to a logfile for example.

this won’t stop logstash (i believe filter failure won’t stop logstash), but it will prevent documents with error to be sent to ES

3

If you actually want to exit the logstash process then see here. If you want to drop the events when you get an error you can call event.cancel in the ruby filter.

4

Thanks ptamba.
Although this is not what i am after, but your suggestion gives me an idea to think about it, whether i should exit logstash process or let it go so that other good records can be inserted.

5

Thanks Badger.

I saw your reply in the other post earlier. And i did try that solution but what i get is the following error message:
"Ruby exception occurred: this signal not yet implemented in windows"

This is how i apply it.

ruby {
code => 'require "open3"
data = """ + event.get("message").strip + """
cmd = "python D:/tklim/Documents/ProgrammingFolder/pythonFiles/python_get_parameters_from_logstas.py #{data}"
Open3.popen3(cmd) do |stdin, stdout, stderr, wait_thr|
begin
event.set("process_result", stdout.readline)
rescue StandardError => e
logger.error("Exception: " + "class name: " + e.class.name + " | " + "error message: " + stderr.read)
Process.kill(15, Process.pid)
end
end'
}

But today, i changed the number from 15 to 9, it works. I think, perhaps it is my pc setting that caused the 'signal not yet implemented in windows' error.

Process.kill(9, Process.pid)

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.