Creating separate elasticsearch docs with batched log messages

I have searched for this and have found a few forum entries that can help me figure out how to get the batch msgs (below) separated but I have no idea what to do with them after they are separated.

As an example, here is an RFC5464 syslog message. The header is easily parsed but the last field (MSG) can be 1 to 3 "\n" separated CSV entries (below has 2 for brevity):

<14>1 2021-09-16T21:48:05Z server pen-netagent 1234 - - 1,2,3,4,5\n6,7,8,9,0

What I need to do in logstash is have the pertinent header info in each document (up to the 2 dashes) and each document would have entries for the CSV info batched in the message. First document would have 1,2,3,4,5 and the second would have 6,7,8,9,0

If I only have 1 CSV MSG in the log, it's easy to parse using dissect (twice, once for the message itself and then again for the CSV part) but I can't figure out how to say:

Cut the MSG into parts based on the "\n" delimiter.
IIterate through each CSV and create a doc for each with the same header info and the current CSV data we are looking at.  
Store these multiple docs into elasticsearch.


You can do the first with mutate+split, and do the second with a split filter.

Thanks!! Will take a look at those.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.