Can I attach a messages byte-count as a metadata field?

Elastic Stack Logstash
1

Hi; I'm currently running a Logstash server in which I receive messages from various sources. I'd like to find out the the approximate byte-count of each incoming message and attach this as a @metadata field, so I can drop excessively large messages, and monitor the approximate size of incoming messages.

Is this possible to implement using the ruby filter? I tried

code => "event['message_size'] = event.message.length

but it didn't work, as message seems to be an object not a string.

Any help would be appreciated.

2 
code => "event['message_size'] = event.message.length

Change to:

code => "event['message_size'] = event['message'].length"
1 Like
3

Thanks for the reply; I tried that variant of the code in my filter block

ruby {
	code => "event['message_size'] = event['message'].length"
}

but each message was labelled with a _rubyexception tag, and no message length field.

Just to note, I'm using the version of Logstash that is released on the Ubuntu PPA

4

Your logs should contain more information about the Ruby exception.

5

Thanks, I didn't actually know Ruby exceptions were stored in logstash.log

{:timestamp=>"2016-09-01T15:48:49.535000+0100", :message=>"Ruby exception occurred: undefined method `length' for nil:NilClass", :level=>:error}

I get an error essentially saying that the event-message is null, which is strange (I hope this isn't turning into a bug report!)

6

This indeed indicates that the message field is unset.

1 Like
7

I can't believe the cause of the error was this simple; I just added a check that the message is defined, and everything worked perfectly.

if event['message']
	event['message_size'] = event['message'].length
end

I appreciate the help fixing this problem