Dynamic fields in CSV to be pushed to ES

Elastic Stack Logstash
2

A csv filter can only handle one set of column names, if you have a file per month and each file has columns for the following 12 months then I would use ruby.

Use a multiline codec to consume each file as a single event. Something like

file {
    path => "/tmp/foo/?.csv" 
    sincedb_path => "/dev/null" 
    start_position => beginning 
    codec => multiline { 
        pattern => "^Spalanzani" 
        negate => true 
        what => previous 
        auto_flush_interval => 2
    }
}

Then in a ruby filter...

    ruby {
        code => '
            lines = event.get("message")
            lines = lines.gsub(/ *\| */, "|")
            lines = lines.split("\n")

            columns = CSV.parse_line(lines.shift, :col_sep => "|")
            columns.shift   # Discard leading |
            columns.pop     # Discard trailing |

            lines.shift # Discard the row of dashes

            lines.each { |x|
                values = CSV.parse_line(x, :col_sep => "|")
                values.shift # Discard leading |
                columns.each_index { |i| event.set(columns[i], values[i]) }
            }
        '
    }