Dealing with multiple number of values from the same input - Logstash

Elastic Stack Logstash
#1

I have recently started using Logstash so still new at this. The use-case I am working on is that I want to ship a log file with Logstash that has different lengths of lines in it. The length of line (the number of attributes to be extracted) depends on a FLAG which is added right at the start of every line. To elaborate further, I have three flags and the log file contains lines similar to following:

FLAG1, TIMESTAMP, VAL1, VAL2, VAL3, VAL4, VAL6, VAL7, VAL8, VAL9
FLAG2, TIMESTAMP, VAL1, VAL2, VAL3, VAL4, VAL6, VAL7
FLAG3, TIMESTAMP, VAL1, VAL2, VAL3, VAL4, VAL6

Can someone explain how can I handle this in my filter section of config file?

#2

I would do something like this.

If all types of rows have the same fields (and FLAG2/FLAG3 rows are missing some) then you can use a single csv filter with the ignore_empty_columns set to true.

#3

Thanks for the response, @Badger.

But it seems like else block is not being running. Just a clarifying question, when we count the number of fields from metadata, does the code count number of fields in the CSV file or in the particular record?

#4

P.S. rows followed by different flags contain different fields altogether i.e. I will have to give them different column names after checking their flags or number of columns.

#5

It counts it for each event (line), not for the file.

1 Like
#6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.