Logstash ruby filter issues with HTTP queries

Elastic Stack Logstash
1

Hi there!
I'm stuck with the issue, I have with an http client in my ruby filter in Logstash 6.5.4.

Here is a snippet of the filter code

require 'net/http'
require 'uri'
require 'json'

def register(params)
  @http = Net::HTTP.new("http://dcaserver", 8080)
  @log = Logger.new("xfecats.log", 5, 10240000)
  @log.level = Logger::DEBUG
end

def filter(event)
...
        begin
          response = @http.request ( Net::HTTP::Get.new("/url/mobile/#{domain}") )
          if response.is_a? Net::HTTPOK
            json_string = response.body
            json = JSON.parse(json_string)
            json_cats = json['value']
            if json_cats.nil?
              new_event.set("[xfe][category]", "Unknown")
              return[new_event]
            end
            ...
        rescue Exception => ex
          @log.debug "Exception querying DCA for domain #{domain} :#{ex.message}\n#{ex.backtrace}"
        end
        return[new_event]
.....
end

So, basically, my filter makes HTTP request to some local service. There are could be many of them. What I noticed, that almost 90% of them are failing with strange errors, like these three:

D, [2019-05-07T14:39:39.077899 #1] DEBUG -- : Exception querying DCA for domain google.cf :undefined method `+' for nil:NilClass
["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/protocol.rb:212:in `write0'",
"uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/protocol.rb:185:in `block in write'",
"uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/protocol.rb:202:in `writing'",
"uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/protocol.rb:184:in `write'",
"uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http/generic_request.rb:334:in `write_header'",
"uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http/generic_request.rb:127:in `exec'",
"uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1446:in `block in transport_request'",
"org/jruby/RubyKernel.java:1114:in `catch'",
"uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1445:in `transport_request'",
"uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1418:in `request'",
"uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1411:in `block in request'",
"uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:858:in `start'",
"uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1409:in `request'",
"/usr/share/logstash/xfecategories.rb:127:in `block in filter'", "org/jruby/RubyArray.java:1734:in `each'",
"/usr/share/logstash/xfecategories.rb:97:in `filter'",
"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-filter-ruby-3.1.5/lib/logstash/filters/ruby/script/context.rb:55:in `execute_filter'",...

D, [2019-05-07T14:39:39.327289 #1] DEBUG -- : Exception querying DCA for domain qtnet.ad.jp :wrong status line: "Access-Control-Allow-Origin: *"
["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http/response.rb:42:in `read_status_line'",
"uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http/response.rb:29:in `read_new'",
"uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1448:in `block in transport_request'",
"org/jruby/RubyKernel.java:1114:in `catch'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1445:in `transport_request'",
"uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1418:in `request'",
"uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1411:in `block in request'",
"uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:858:in `start'",
"uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1409:in `request'",
"/usr/share/logstash/xfecategories.rb:127:in `block in filter'", "org/jruby/RubyArray.java:1734:in `each'",
"/usr/share/logstash/xfecategories.rb:97:in `filter'",...

D, [2019-05-07T14:39:39.180247 #1] DEBUG -- : Exception querying DCA for domain ghnet.com.br :undefined method `closed?' for nil:NilClass
Did you mean?  clone
["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1486:in `begin_transport'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1444:in `transport_request'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1418:in `request'", "/usr/share/logstash/xfecategories.rb:127:in `block in filter'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/xfecategories.rb:97:in `filter'", ...

After some googling, I figured out that these issues might be connected the use of WebMock gem, see this. But I don't intentially use WebMock and I can't see that Logstash does.

Does anyone have any piece of advice?
I'm completely stuck here...

Thanks!

2

Does setting "--pipeline.workers 1" eliminate the errors? If it does you have thread-safety issues.

3

It did help.

Thanks a lot!

4

BTW, you know there is an http filter plugin, right?

5

No, I didn't. Thanks for pointing out!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.