Using ruby to clean part of a message

Elastic Stack Logstash
1

Hi All,

We are trying to send messages from an application server to Kafka, but some rows are too long so that we have to get rid of some part of the row. We can filter out long messages by ruby but we need to process further not to loose these messages. An example row is as below;

2018-05-28 01:58:39.993||abc1||10.251.55.22||10.251.55.22||RC_USER||-||service_abc_execute||1||{"servisAdi":"kaydet_xyz","islemOid":"0mjhoj3k4i1brt","veriler":[],"vtAdi":"57"}$$ISLEMSURESI=1

We want to clean the part contained in {} but push all other information to Kafka. Is it possible to do this in Ruby?

Thanks

2

You can e.g. use a mutate filter and its gsub option to delete everything between two braces, but if braces can occur elsewhere in the log messages that might not be a great idea.

Another option is to use a csv filter (with || as the separator) to separate the string into fields, use e.g. a mutate filter to process the field with the JSON string, and then piece everything back again (unless you want to send a JSON object to Kafka).

3

The exact problem is we don't want to parse every event but only the ones which are long. So we used ruby filter to find these long messages. But we couldn't go further with ruby. It is a costly operation to parse every event on our side. So we need to parse only the long events (which are filtered).

4

Oh, right. Well, the same methodologies that I outlined can of course be done from Ruby code.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.