Multiple CSV files and Multiple Header data [identifiers]

[theuntergeek (Aaron Mildenstein)]

You're referring to this post:

@theuntergeek

The content is always split into lines, but without a conditional, the csv filter is totally unaware of which line of a file it is receiving. This is why a conditionals are essential, and knowing what the possible column types are.

if [csv_type_1] {
  if [message] =~ /headerpattern/ {
    csv { ... }
  }
}

csv_type_1 identifies a stream (by file type, or whatever you identify it with), headerpattern will be the way you can tell the first line is a header and not data, and then you apply the known csv column match to the data.

Logstash 2.x will simplify this somewhat as we are designing it to allow for multiple pipelines. A (hypothetical) CSV pipeline would allow for ingesting a single file, and using the first line to define column/field names. In the meanwhile, unfortunately there is nothing in Logstash which will do what you are asking.

Included here for quick access.

The idea behind csv_type_1 is arbitrary. It just needs to be a way you can concretely identify the source of the data to differentiate it from other sources of data. If you only have a single source of data, this conditional is unnecessary.