[ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined method `split' for nil:NilClass

[mattsdevop]

Attempting to send syslog from a Barracuda firewall to logstash. Hoping somebody can help out here. Using logstash configuration from here (waf.conf) with the appropriate changes for my stack: https://campus.barracuda.com/product/webapplicationfirewall/doc/70584459/integrating-the-elk-stack-with-the-barracuda-web-application-firewall/

I've configured the custom log format in the Barracuda according to the article. When I setup the syslog server information for my logstash endpoint, the logstash log fills with
[ERROR][logstash.filters.ruby ] Ruby exception occurred: undefined method `split' for nil:NilClass
and the index is never created.

The relevant Ruby code is:

	init => "
	        HEADER_FIELDS = ['cef_version','DeviceVendor','Product','DeviceVersion','SignatureID','Name','Severity']
	      	#event_new = LogStash::Event.new
		      def store_header_field(event,field_name,field_data)
		          #Unescape pipes and backslash in header fields
		          event.set(field_name,field_data.gsub(/\\\|/, '|').gsub(/\\\\/, '\\')) unless field_data.nil?
		      end
		
      	"
	code => "
	        if event.get('[message][0]') == '\"'
		         event.set('[message]' , event.get('[message]')[1..-2])
		      end
		      split_data = event.get('[message]').split /(?<=[^\\]\\\\)[\|]|(?<!\\)[\|]/
      		
      		HEADER_FIELDS.each_with_index do |field_name, index|
      		   store_header_field(event,field_name,split_data[index])
      		end
          msg = split_data[HEADER_FIELDS.size..-1].join('|')
      		if event.get('cef_version').include? ' '
      		   split_cef_version= event.get('cef_version').rpartition(' ')
      	           event.set('syslog', split_cef_version[0])
      	           event.set('cef_version',split_cef_version[2])
      	        end
      		event.set('cef_version', event.get('cef_version').sub(/^CEF:/, ''))
     		  if not msg.nil? and msg.include? '='
      		   msg = msg.strip
             
      		   # If the last KVP has no value, add an empty string, this prevents hash errors below
             if msg.end_with?('=')
      		      msg=msg + ' ' unless msg.end_with?('\=')
	           end

	           # Now parse the key value pairs into it
	           msg = msg.split(/ ([\w\.]+)=/)
	           key, value = msg.shift.split('=', 2)
             event.set(key,value.gsub(/\\=/, '=').gsub(/\\\\/, '\\'))
             Hash[*msg].each{ |k, v| event.set(k,v.gsub(/\\=/, '=').gsub(/\\\\/, '\\')) unless v.nil? }
	          
           end
	        "
  remove_field => ['message']
  }```

[Badger]

@mattsdevop

msg = msg.split(/ ([\w.]+)=/) key, value = msg.shift.split('=', 2)

It seems either msg is nil, or msg.shift is nil. To debug it I suggest you run logstash with a generator input that contains a [message] that triggers this and a stdout output. Then start using puts in your filter to see what the values are.

[mattsdevop]

I appreciate your reply. I'm currently working with Barracuda in hopes that they have already done the legwork, and can provide the fixed code. If they do, I will update. If they aren't able to, and I get around to being able to debug with your suggestion, I'll be sure to update as well.

[mattsdevop]

I was able to view the dump from the Barracuda today. It does not appear the logs are in the format that the Ruby code is expecting from a cursory look. I've forwarded the dump to Barracuda to continue working toward the solution. Just wanted to update here in case anyone else lands on this same issue.

[mattsdevop]

I've made progress on my own. In a testing environment, I turned on debug mode for logstash. This led me to discover that logstash doesn't discriminate on which logs the Ruby code acts on. So, if you have an existing logs that you ingest, they will need to be tagged to handle separately from existing ingesting. Or, the existing configuration will need to be removed and replaced with theirs (breaking your existing configuration).
Using the files they provide on their site, I was able to finally ingest some logs and create indices in my test environment. Some of the information being passed to logstash is producing a 'join' error now. The debug logs did not lend much insight into what's going on here. I am currently awaiting an update from Barracuda.

[sfio]

Hi @mattsdevop ,

is there anything new on this topic.
I´m also trying to get this to work, but I can´t get logs from the Barracude.
My problem is, that I can´t get port 1514 or 514 to listen status, did you have had similar problems?
I followed the guide and installed the old version which is suggested in the how-to.
Would be nice if they had a config which works with EK 7.

[sfio]

OK it was a layer 8 problem.
Instead of sending messages in udp I tested the connection with telnet which is tcp.
I sended some messages via socat and it worked well.
But somehow I can´t get messages from Barracuda.
In Firewall I opened all UDP ports but theres nothing arriving.
Could you please giva me a hint, how to get messages from Barracuda?

Here is my config:

[mattsdevop]

On your server running logstash, check the logstash logs to see if there are any errors. If there are no errors, you may want to turn off your software firewall (firewalld on RHEL distro) on the server while testing. You may also want to check that selinux is not interfering by disabling it (if you are running a RHEL distro).

I hope those tips help. If not, open a new post with your configurations and the issue you are experiencing. We will be able to help you in your separate post. Good luck @sfio

[mattsdevop]

Ultimately, for my issue of errors in the logs from the Barracuda, they responded after their engineering team reviewed the issue. Their response is that there are multiple issues related to the parsing of connection log format and some of the syslogs. They are stating that they will have this resolved in a future firmware update.

[sfio]

Hi @mattsdevop,

thank you for the reply.
It was in fact a firewall (issue), but not on the operating system (Debian 9), there was another fw in between of ES and WAF, which I didn't know of. After passthrough the packets it's working.
Just getting error for the firewall logs from WAF, but that is another topic I think.
Do you know if Barracuda will bring out templates for ES 7, or do you know how to convert them?

BR