Ruby code include?

Elastic Stack Logstash
1

Hi All,

I have this code that used to be working in ELK 7.12 now that I've upgrade to 8.7.1 it gives a weird error in logstash

             code =>
             "

                ip_src = Array.new
                ip_src.push(IPAddr.new(event.get('OSA'),Socket::AF_INET))


                i = 0
                found = 0
                @network_address.product(ip_src).each do |n, src|
                  if found > 1
                        return
                  end
                  if n.include?(src)
                    #event.set('[src_network_address]', n.to_s)
                    event.set('src_network_location', @network_location[i])
                    event.set('src_network_name', @network_name[i])
                    found += 1
                  end
                  i += 1
                end
            "

The error is

[ERROR] 2023-08-08 15:54:06.163 [[main]>worker7] ruby - Ruby exception occurred: undefined method `include?' for 2:Integer {:class=>"NoMethodError", :backtrace=>["(ruby filter code):14:in `block in filter_method'", "org/jruby/RubyArray.java:1865:in `each'", "(ruby filter code):10:in `block in filter_method'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:96:in `inline_script'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:89:in `filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159:in `do_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:178:in `block in multi_filter'", "org/jruby/RubyArray.java:1865:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:175:in `multi_filter'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:133:in `multi_filter'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304:in `block in start_workers'"]}

For some reason its complaining that the line with n.include? is unknown.
Is there a new syntax for this?
OSA= Original Source Address (its a v4 IP)
network_address is an array of IP/netwmask

Thanks for your help.

Regards,

Michael

2

That suggests that the first entry in @network_address is an integer (with value 2). You don't show how that gets initialized.

3

Thanks for your response Badger.

Full code looks like:

input {
    tcp {
        port => 1514
        codec => multiline{
                pattern => "^\["
                what => "previous"
                negate=> true
        }
    }
    udp {
        port => 1514
        codec => multiline{
                pattern => "^\["
                what => "previous"
                negate=> true
        }
    }
}

filter{
  grok{
    match => ['message',"<166>%{PROG:program} TM=%{BASE16FLOAT:tm} IF=%{DATA:IF} OF=%{DATA:OF} IFS=%{DATA:IFS} UID=%{DATA:UUID} BID=%{DATA:BID} MAC=%{MAC:MAC} PRO=%{DATA:PRO} OSA=%{DATA:OSA}:%{INT:OSP} ODA=%{DATA:ODA} SA=%{DATA:SA} DA=%{DATA:DA}:%{DATA:DAPORT} MET=%{DATA:MET} HOST=%{DATA:HOST} UA=%{DATA:UA} URI=%{GREEDYDATA:URI}"]
    match => ['message',"<166>%{SYSLOGTIMESTAMP:date} %{DATA:server} %{PROG:program} TM=%{BASE16FLOAT:tm} IF=%{DATA:IF} OF=%{DATA:OF} IFS=%{DATA:IFS} UID=%{DATA:UUID} BID=%{DATA:BID} MAC=%{MAC:MAC} PRO=%{DATA:PRO} OSA=%{DATA:OSA}:%{INT:OSP} ODA=%{DATA:ODA} SA=%{DATA:SA} DA=%{DATA:DA}:%{DATA:DAPORT} MET=%{DATA:MET} HOST=%{DATA:HOST} UA=%{DATA:UA} URI=%{GREEDYDATA:URI}"]
  }
         if [DA] {
          geoip {
                 source => "DA"
                 ecs_compatibility => disabled
                }
         }
         if [UA] != ""  {
                useragent {
                        add_tag => [ "UA" ]
                        source => "[UA]"
                    }
         }


        ruby
        {
             init =>
             "
                require 'csv'
                @network_address = Array.new()
                @network_location = Array.new()
                @network_name = Array.new()

                CSV.foreach('/data/subnet2.csv' , col_sep: ',') do |row|
                  @network_address.push(IPAddr.new(row[0]),Socket::AF_INET)
                  @network_location.push(row[1])
                  @network_name.push(row[2])
                end
             "
             code =>
             "

                ip_src = Array.new
                ip_src.push(IPAddr.new(event.get('OSA'),Socket::AF_INET))


                i = 0
                found = 0
                @network_address.product(ip_src).each do |n, src|
                  if found > 1
                        return
                  end
                  if n.include?(src)
                    #event.set('[src_network_address]', n.to_s)
                    event.set('src_network_location', @network_location[i])
                    event.set('src_network_name', @network_name[i])
                    found += 1
                  end
                  i += 1
                end
            "
  }
  if "_grokparsefailure" in [tags] {
     drop { }
  }
}
output {
    stdout {
        codec => rubydebug
    }
}

Left out the output elasticsearch section.

The csv looks like
10.10.0.0/16,Office,vlan10
192.168.123.0/24,Spartan,Spartan

The log messages looks something like

TM=0.0 IF=eth0.10 OF=eth1 IFS=eth0.10 UID=h7cpj2,1 BID=michael.van MAC=9C:FC:E8:87:A1:BD PRO=6 OSA=10.10.6.251:61498 ODA=23.50.29.210:443 SA=10.10.6.251:61498 DA=23.50.29.210:443 MET=OTHER HOST=osce14-0-en.url.trendmicro.com UA= URI=

TM=0.0 IF=eth0.10 OF=eth1 IFS=eth0.10 UID=h7cpj2,1 BID=michael.van MAC=9C:FC:E8:87:A1:BD PRO=6 OSA=10.10.6.251:61497 ODA=111.223.64.80:443 SA=10.10.6.251:61497 DA=111.223.64.80:443 MET=OTHER HOST= UA= URI=
4

I think the ,Socket::AF_INET should be inside the .new(), or you could just remove it. Socket::AF_INET is a constant with the value 2, so @network_addresses ends up as an array with 4 entries:

[#<IPAddr: IPv4:10.10.0.0/255.255.0.0>, 2, #<IPAddr: IPv4:192.168.123.0/255.255.255.0>, 2]

When you do the .product.each that ends up doing 2.include?, which throws the exception you are seeing.

If you change that to @network_address.push(IPAddr.new(row[0])) and adjust the sample messages so that they match your groks then you will get events like

{
                 "MET" => "OTHER",
    "src_network_name" => "vlan10",
             "message" => "TM=0.0 IF=eth0.10 OF=eth1 IFS=eth0.10 UID=h7cpj2,1 BID=michael.van MAC=9C:FC:E8:87:A1:BD PRO=6 OSA=10.10.6.251:61497 ODA=111.223.64.80:443 SA=10.10.6.251:61497 DA=111.223.64.80:443 MET=OTHER HOST= UA= URI=",
               "geoip" => {
    "continent_code" => "AS",
          "location" => {
        "lon" => 103.8547,
        "lat" => 1.2929
    },
          "timezone" => "Asia/Singapore",
    ....
}
5

Hi Badger,

When I remove the Socket::AF_NET, I get this error.

[ERROR] 2023-08-11 11:12:46.506 [[main]>worker5] ruby - Ruby exception occurred: address family must be specified {:class=>"IPAddr::AddressFamilyError", :backtrace=>["/usr/share/logstash/vendor/jruby/lib/ruby/stdlib/ipaddr.rb:565:in `initialize'", "org/jruby/RubyClass.java:890:in `new'", "(ruby filter code):5:in `block in filter_method'", "/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:96:in `inline_script'", "/usr/sb/logstash/filters/base.rb:178:in `block in multi_filter'", "org/jruby/RubyArray.java:1865:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:175:in `multi_filter'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:133:in `multi_filter'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304:in `block in start_workers'"]}

In elk/logstash 7.x this wasn't a problem, in 8.6 it is a problem.

Regards,
Michael

6

OK, so move it inside the .new(). I cannot find what version of ruby makes this a mandatory argument, but setting it should be OK.

7

Thanks Badger,

Had to tweak the grok pattern because it was a relayed message.

But it now works.

Thanks for your help.

Regards,

Michael

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.