# Dynamic fields in CSV to be pushed to ES

**URL:** <https://discuss.elastic.co/t/dynamic-fields-in-csv-to-be-pushed-to-es/355892>\
**Category:** Logstash\
**Created:** [March 21, 2024, 10:31am UTC](https://discuss.elastic.co/t/dynamic-fields-in-csv-to-be-pushed-to-es/355892 "2024-03-21T10:31:12Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 21, 2024, 7:00pm UTC](https://discuss.elastic.co/t/dynamic-fields-in-csv-to-be-pushed-to-es/355892/2 "2024-03-21T19:00:02Z")

</div>

A csv filter can only handle one set of column names, if you have a file per month and each file has columns for the following 12 months then I would use ruby.

Use a multiline codec to [consume each file as a single event](https://discuss.elastic.co/t/parsing-array-of-json-objects-with-logstash-and-injesting-to-elastic/203197/2). Something like

```
file {
    path => "/tmp/foo/?.csv" 
    sincedb_path => "/dev/null" 
    start_position => beginning 
    codec => multiline { 
        pattern => "^Spalanzani" 
        negate => true 
        what => previous 
        auto_flush_interval => 2
    }
}

```

Then in a ruby filter...

```
    ruby {
        code => '
            lines = event.get("message")
            lines = lines.gsub(/ *\| */, "|")
            lines = lines.split("\n")

            columns = CSV.parse_line(lines.shift, :col_sep => "|")
            columns.shift # Discard leading |
            columns.pop # Discard trailing |

            lines.shift # Discard the row of dashes

            lines.each { |x|
                values = CSV.parse_line(x, :col_sep => "|")
                values.shift # Discard leading |
                columns.each_index { |i| event.set(columns[i], values[i]) }
            }
        '
    }

```

---

_[View the full topic](https://discuss.elastic.co/t/dynamic-fields-in-csv-to-be-pushed-to-es/355892)._