Multiple CSV files and Multiple Header data [identifiers]


(Mike W) #1

I have read @theuntergeek responses in the Parse 1st Line of Multiple CSV files and set as Columns thread

But I didn't fully understand this comment:

csv_type_1 identifies a stream (by file type, or whatever you identify it with)

Can I have an example please? Where do I define csv_type_1?


(Aaron Mildenstein) #2

You're referring to this post:

Included here for quick access.

The idea behind csv_type_1 is arbitrary. It just needs to be a way you can concretely identify the source of the data to differentiate it from other sources of data. If you only have a single source of data, this conditional is unnecessary.


(Mike W) #3

Thanks, I understand it, but I don't know how can I tag specific stream as csv_type_1 or anything else. Maybe I'm overthinking it :wink:


(Walker) #4

Can you post your input so we can see how you're ingesting data?

Basically, @theuntergeek is running two logic checks, one inside the other. Basically, it says:

IF the field, `csv_type_1` exists in the event {
  IF the field, `message` matches regular expression pattern `headerpattern` {
    perform csv parsing { ... }
  }
}

Both csv_type_1 and message are fields inside the event. Message will always exist because thats where logstash sticks the raw data it receives. Csv_type_1 is a field that he just came up with as an example or exists in his own dataset. Unless you are pulling logs from the same type of device, you are going to use a different field to qualify the statement.


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.