Logstash and Ruby

Elastic Stack Logstash
1

Hello,

I am trying to integrate a Ruby script into Logstash, but I am having a hard time... Basically, I have a field which I want to modify in Ruby and then give back to Logstash. What I'm having a hard time with is the communication between Logstash and Ruby.

For the record, here are the parts I use:

Logstash:
csv {
columns => [ "FUTURE_USE", "temp Receive time", "Serial number", "Type", "Subtype", "FUTURE_USE2", "temp Generated time", "Source IP", "Destination IP", "NAT source IP", "NAT destination IP", "Rule name", "Source user","Destination user", "Application", "Virtual system",
"Source zone", "Destination zone", "Source interface", "Destination interface", "Log forwarding profile", "temp Log creation time", "Session ID", "Repeat count", "Source port", "Destination port", "NAT source port", "NAT destination port", "Flag", "Protocol", "Action",
"Miscellaneous", "Thread ID", "Category", "Severity", "Direction", "Sequence number", "Action flags", "Source location", "Destination location", "FUTURE_USE3", "Content type", "PCAP ID", "Filedigest", "Cloud", "URL index", "User agent", "File type", "X-forwarded-for", "Referer", "Sender",
"Subject", "Recipient", "Report ID",
"Device Group Hierarchy Level 1", "Device Group Hierarchy Level 2", "Device Group Hierarchy Level 3", "Device Group Hierarchy Level 4", "Virtual system name", "Device name", "Action source", "FUTURE_USE4"]
}
ruby {
path => "/etc/logstash/conf.d/test.rb"
script_params => { "flag" => flag }
}

Ruby:
def register(params)
@flag = params["flag"]
end
def filter(event)
flags = {"PCAP" => 0x80000000, "IPV6" => 0x02000000 , "SSL Proxy" => 0x01000000 , "URL Filtering" => 0x00800000, "NAT" => 0x00400000 , "Captive Portal" => 0x00200000 , "X-Forwarded-For" => 0x00080000 , "Proxy Transaction" => 0x00040000 , "Container Page" => 0x00008000, "session has a temporary match on a rule for implicit application dependency handling" => 0x00002000, "symmetric return was used to forward traffic for this session" => 0x00000800 }
flags_to_return = Array.new()
flags.each do |key, value|
result = @flag & value
if value == result
flags_to_return.push(key)
end
end
return flags_to_return
end

2

The ruby script should return [event], unless it wants to drop the event, in which case it should return [], or add new events, in which case it should return an array of events.

If you want to add a field to the event then use event.set("fieldname", value), as described in the event api documentation. So I think you are going to want to do

event.set("flags", flags_to_return)
return [event]
3

Thanks for you help, I didn't fully grasped the concept of events, but this helped.
I advanced on my code, but it seems the if is never true.

def register(params)
		@flag = params["flag"]
end
def filter(event)
		flags = {"PCAP" => 0x80000000, "IPV6" => 0x02000000 , "SSL Proxy" => 0x01000000 , "URL Filtering" => 0x00800000, "NAT" => 0x00400000 , "Captive Portal" => 0x00200000 , "X-Forwarded-For" => 0x00080000 , "Proxy Transaction" => 0x00040000 , "Container Page" => 0x00008000,  "session has a temporary match on a r$
		flags_to_return = Array.new()
		flags.each do |key, value|
				if @flag.hex & value == value
						flags_to_return.push(key)
				end
		end
		event.set("flag", flags_to_return.join(" "))
		return [event]
end

If I test the script, setting flag with a mere string containing a hex code (ie. "0x60000"), it works, but with Logstash, it doesn't seem to take...

4

I'm not sure what you're trying to do here. The way you've written the code the contents of flags_to_return is independent of what's in the event. That's either a bug or you can pre-compute the value of the flag field in the register function.

5

Hi there thanks for your help, but it is not independent from the flag value, thanks to the 8th line:
if @flag.hex & value == value

6

Yes, as I said it's dependent of the flag value but that value is independent from the event being processed so you're overcomplicating things. You can just stuff all the mappings from the flags hash in a file and use a translate filter.

7

Mmmhhh... Didn't know this one plugin. But I cannot do an "and" with the translate filter, right?

8

I don't think you're getting the point I'm trying to make. Do you want your filter to inspect the Flag field of each event and based on its contents populate some other field with a list of strings like "PCAP", "IPV6", ...?

9

Yes. I thought that was what I was doing....:sweat_smile:

10

No. You access fields in events with event.get and event.set. Your current register function serves no purpose. This probably works:

def filter(event)
  flags = {"PCAP" => 0x80000000, "IPV6" => 0x02000000 , "SSL Proxy" => 0x01000000 , "URL Filtering" => 0x00800000, "NAT" => 0x00400000 , "Captive Portal" => 0x00200000 , "X-Forwarded-For" => 0x00080000 , "Proxy Transaction" => 0x00040000 , "Container Page" => 0x00008000}
  flags_to_return = []
  flags.each do |key, value|
    if event.get("flag").hex & value == value
      flags_to_return.push(key)
    end
  end
  event.set("flag", flags_to_return)
  return [event]
end

(There's room for optimization if you have a high event rate.)

11

The funny thing is, that what I was trying to do (didn't update my code here). In the end, the problem was (obviously), a dumb one: a typo. Thank you very much for your help!

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.