# Can I use an existing field in the event to create the columns for a CSV filter?

**URL:** <https://discuss.elastic.co/t/can-i-use-an-existing-field-in-the-event-to-create-the-columns-for-a-csv-filter/244258>\
**Category:** Logstash\
**Created:** [August 8, 2020, 4:32pm UTC](https://discuss.elastic.co/t/can-i-use-an-existing-field-in-the-event-to-create-the-columns-for-a-csv-filter/244258 "2020-08-08T16:32:25Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![crickes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/crickes/32/18009_2.png) [@crickes](https://discuss.elastic.co/u/crickes)\
**Post date:** [August 8, 2020, 4:32pm UTC](https://discuss.elastic.co/t/can-i-use-an-existing-field-in-the-event-to-create-the-columns-for-a-csv-filter/244258/1 "2020-08-08T16:32:26Z")

</div>

I've been struugling with this for far too long and I'm giving up and asking for help. I am being sent data via a Kafka topic which contains a field called 'headers' which is a CSV string of column headers, and a field called 'data' which is a CSV string of the corresponding data. What I want to achieve is to be able to use the header string as the 'columns' option is a CSV filter.

I am trying to use this but so far unsuccessfully:

```auto
csv {
        source => "Data"
        columns => "%{Headers}"
        target => "data"
      }

```

I have tried using,

```auto
mutate {
      split => { "headers" => ","}
     }

```

first, to create an array of headers, but this doesn't seem to work either as in the output I end up with 2 arrays, one of headers:

```auto
    "Headers" => [
            [0] "serverIP",
            [1] "statsType",
            [2] "adjacencyName",
            [3] "adjStatAccountName",
            [4] "adjStatSrcActiveCalls",
            [5] "adjStatDstActiveCalls",
            [6] "adjStatSrcActivatingCalls"
    ]

```

and one for the data:

```auto
"data" => [
             "column168" => "0",
             "column245" => "0",
              "column18" => "0",
             "column209" => "4294967295",
             "column244" => "0",
             "column153" => "0 / 10000 %"
]

```

In reality, the headers and corresponsing data strings have several hundred fields in them, so I don't really want to have to write them all out in the logstash filter.

Can anyone suggest a way that I can build the columns array in such a way that I can then pass it to the CSV filter to be used so that I get a proper key =\> value pair result?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 8, 2020, 5:00pm UTC](https://discuss.elastic.co/t/can-i-use-an-existing-field-in-the-event-to-create-the-columns-for-a-csv-filter/244258/2 "2020-08-08T17:00:13Z")

</div>

I would do that in ruby

```
input { generator { count => 1 lines => ['{ "headers": "a,b,c,d", "data": "1,2,3,4" }', '{ "headers": "d,e,f,g", "data": "9,8,7,6" }'] } }
filter {
    json { source => "message" remove_field => ["message"] }
    ruby {
        init => "
            require 'csv'
        "
        code => '
            data = event.get("data")
            headers = event.get("headers")
            if headers and data
                table = CSV.parse(data, { :headers => headers })
                dataAsHash = table.map { |row| row.to_hash }
            end
            # event.set("dataAsHash", dataAsHash[0])
            # or
            dataAsHash[0].each { |k, v| event.set(k, v) }
        '
    }
}

```

will get you

```
{
"@timestamp" => 2020-08-08T16:56:42.797Z,
      "data" => "1,2,3,4",
         "a" => "1",
         "b" => "2",
         "c" => "3",
         "d" => "4",
   "headers" => "a,b,c,d",
...

```

You might want to add

```
remove_field => ["data", "headers"]

```

to the ruby filter so that those fields are removed if there are no ruby exceptions.

---

<div class="post-metadata">

**Author:** ![crickes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/crickes/32/18009_2.png) [@crickes](https://discuss.elastic.co/u/crickes)\
**Post date:** [August 8, 2020, 5:10pm UTC](https://discuss.elastic.co/t/can-i-use-an-existing-field-in-the-event-to-create-the-columns-for-a-csv-filter/244258/3 "2020-08-08T17:10:27Z")

</div>

Thank you ! Worked perfectly.

I did try to use Ruby to try stich the header string and data string together, but alas that didn't work. This though is perfect. Thanks.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 5, 2020, 5:10pm UTC](https://discuss.elastic.co/t/can-i-use-an-existing-field-in-the-event-to-create-the-columns-for-a-csv-filter/244258/4 "2020-09-05T17:10:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.