Logstash 5 ruby filter

[pjanzen (Paul Janzen)]

Hi All,

I have the following code in my logstash config and the ruby part causes logstash to continuously restart, if I remove the ruby part it works just fine. There are no errors in the log what so ever, shall I report a bug for this? Or is there something I do wrong here?

        if [message] =~ /^<=/ {
            dissect {
                mapping => {
                    "message" => "%{exim_flag} %{enveloppe_sender_address} %{message}"
               }
            }
            grok {
                match => {
                    "message" => ".+ \[(?<ip>\d{1,3}.\d{1,3}.\d{1,3}.\d{1,3})\] .+ S=(?<size>\d+) .+ T=(?<subject>.+) for (?<recipients>.+)"
                }
            }

            mutate {
                replace => { "exim_flag" => "arrival" }
            }

            if [subject] =~ /utf-8/ {
                ruby {
                    init => "require 'mail'"
                    code => "event.set('subject_decoded') =Mail::Encodings.value_decode(event.get('subject'))"
                }
            }
        }

root@logstash1:/opt/logstash/logs# /usr/share/logstash/bin/logstash -V
logstash 5.2.1

Plugins:
logstash-codec-cef
logstash-codec-collectd
logstash-codec-dots
logstash-codec-edn
logstash-codec-edn_lines
logstash-codec-es_bulk
logstash-codec-fluent
logstash-codec-graphite
logstash-codec-json
logstash-codec-json_lines
logstash-codec-line
logstash-codec-msgpack
logstash-codec-multiline
logstash-codec-netflow
logstash-codec-plain
logstash-codec-rubydebug
logstash-filter-cidr
logstash-filter-clone
logstash-filter-csv
logstash-filter-date
logstash-filter-dissect
logstash-filter-dns
logstash-filter-drop
logstash-filter-fingerprint
logstash-filter-geoip
logstash-filter-grok
logstash-filter-json
logstash-filter-kv
logstash-filter-metrics
logstash-filter-mutate
logstash-filter-ruby
logstash-filter-sleep
logstash-filter-split
logstash-filter-syslog_pri
logstash-filter-throttle
logstash-filter-urldecode
logstash-filter-useragent
logstash-filter-uuid
logstash-filter-xml
logstash-input-beats
logstash-input-couchdb_changes
logstash-input-elasticsearch
logstash-input-exec
logstash-input-file
logstash-input-ganglia
logstash-input-gelf
logstash-input-generator
logstash-input-graphite
logstash-input-heartbeat
logstash-input-http
logstash-input-http_poller
logstash-input-imap
logstash-input-irc
logstash-input-jdbc
logstash-input-kafka
logstash-input-log4j
logstash-input-lumberjack
logstash-input-pipe
logstash-input-rabbitmq
logstash-input-redis
logstash-input-s3
logstash-input-snmptrap
logstash-input-sqs
logstash-input-stdin
logstash-input-syslog
logstash-input-tcp
logstash-input-twitter
logstash-input-udp
logstash-input-unix
logstash-input-xmpp
logstash-output-cloudwatch
logstash-output-csv
logstash-output-elasticsearch
logstash-output-file
logstash-output-graphite
logstash-output-http
logstash-output-irc
logstash-output-kafka
logstash-output-nagios
logstash-output-null
logstash-output-pagerduty
logstash-output-pipe
logstash-output-rabbitmq
logstash-output-redis
logstash-output-s3
logstash-output-sns
logstash-output-sqs
logstash-output-statsd
logstash-output-stdout
logstash-output-tcp
logstash-output-udp
logstash-output-webhdfs
logstash-output-xmpp
logstash-patterns-core

[jkuang (Jimmy Kuang)]

See if you can generate more logging by running Logstash in debug mode.

DEBUG=1 ./logstash....

[pjanzen (Paul Janzen)]

Thanks, that helps a lot... Apparently there is something wrong with this:

event.set('subject_decoded') =Mail::Encodings.value_decode(event.get('subject'))

DEBUG: exec /usr/share/logstash/vendor/jruby/bin/jruby /usr/share/logstash/lib/bootstrap/environment.rb logstash/runner.rb --path.settings /etc/logstash/ -f .
Sending Logstash's logs to /opt/logstash/logs which is now configured via log4j2.properties
SyntaxError: (ruby filter code):1: syntax error, unexpected '='
@codeblock = lambda { |event, &new_event_block| event.set('b64_decoded') = Base64.decode64(event.get('subject')) }
^
eval at org/jruby/RubyKernel.java:1079
register at /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-ruby-3.0.2/lib/logstash/filters/ruby.rb:38
register at /usr/share/logstash/vendor/jruby/lib/ruby/1.9/forwardable.rb:201
start_workers at /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:235
each at org/jruby/RubyArray.java:1613
start_workers at /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:235
run at /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:188
start_pipeline at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:302

Going to do some more digging and fiddeling...

[magnusbaeck (Magnus Bäck)]

event.set('subject_decoded') =Mail::Encodings.value_decode(event.get('subject'))

That's indeed not the correct syntax. Change to:

event.set('subject_decoded', Mail::Encodings.value_decode(event.get('subject')))

See: Event API | Logstash Reference [8.11] | Elastic