Split event into multiple different messages

[jvolpi (Julian)]

Hi everyone

Is there a way to split an event into multiple messages, and send them to elasticsearh?

The problem is the event cannot be splitted in a automatic way using plugins, because the data isn't exactly structured and may vary. The only option is to substring it with 'if' clauses and add new fields as is needed, so the splitted messages will be different between each other.

If there is no way of doing this inside logstash, i guess i will have to pre-process the logs before sending them to logstash.

Thanks in advance

[magnusbaeck (Magnus Bäck)]

If you give an example we can help you.

[jvolpi (Julian)]

Here is a simplified type of message:

msg = 03;00;00;00;00;54;D5;15;DC;CA;FF;05;A7;00;01

As you can see, the messages are just a bunch of bytes separated by semicolons. The thing is that sometimes a group of these bytes represents a single field in elasticsearch, so the spliting needs to be scripted.

In this example i would compare the first byte ('03') to a specific value, if that checks then i would proceed to split the message in the following manner:

msg 1 = 03;00;00;00;00
msg 2 = 54;D5;15;DC;CA
msg 3 = FF;05;A7;00;01

Note: the start and end of each msg may vary, and i can determine those positions using 'if' clauses (the values checked aren't important, they are in each submessage and i only need to look for the 'first byte' of each one).

After splitting the event, i need to assign new fields to each submessage in a scripted way (by using 'if' clauses, processing the bytes inside). All the messages will have the same fields, but the value of each one may be obtained in different ways.

At this point i will have, theoretically, a list of different messages ready to be sent to elasticsearch.

I hope that helps

[magnusbaeck (Magnus Bäck)]

Use a ruby filter to rewrite the input string into an array of strings, then feed that field to a split filter.

[jvolpi (Julian)]

Thank you, it worked for me.