Dealing with multiple number of values from the same input - Logstash

I have recently started using Logstash so still new at this. The use-case I am working on is that I want to ship a log file with Logstash that has different lengths of lines in it. The length of line (the number of attributes to be extracted) depends on a FLAG which is added right at the start of every line. To elaborate further, I have three flags and the log file contains lines similar to following:

FLAG1, TIMESTAMP, VAL1, VAL2, VAL3, VAL4, VAL6, VAL7, VAL8, VAL9
FLAG2, TIMESTAMP, VAL1, VAL2, VAL3, VAL4, VAL6, VAL7
FLAG3, TIMESTAMP, VAL1, VAL2, VAL3, VAL4, VAL6

Can someone explain how can I handle this in my filter section of config file?

I would do something like this.

If all types of rows have the same fields (and FLAG2/FLAG3 rows are missing some) then you can use a single csv filter with the ignore_empty_columns set to true.

Thanks for the response, @Badger.

But it seems like else block is not being running. Just a clarifying question, when we count the number of fields from metadata, does the code count number of fields in the CSV file or in the particular record?

P.S. rows followed by different flags contain different fields altogether i.e. I will have to give them different column names after checking their flags or number of columns.

It counts it for each event (line), not for the file.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.