# Logstash CSV Filter: Define Columns from Event Data/Fields

**URL:** <https://discuss.elastic.co/t/logstash-csv-filter-define-columns-from-event-data-fields/163014>\
**Category:** Logstash\
**Created:** [January 4, 2019, 10:36pm UTC](https://discuss.elastic.co/t/logstash-csv-filter-define-columns-from-event-data-fields/163014 "2019-01-04T22:36:22Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![dm90](https://avatars.discourse-cdn.com/v4/letter/d/e8c25b/32.png) [@dm90](https://discuss.elastic.co/u/dm90)\
**Post date:** [January 4, 2019, 10:36pm UTC](https://discuss.elastic.co/t/logstash-csv-filter-define-columns-from-event-data-fields/163014/1 "2019-01-04T22:36:22Z")

</div>

This is related to: [CSV Filter Column](https://discuss.elastic.co/t/csv-filter-column/122752)

Use case:

The S3 input plugin can return AWS CloudFront logs from an S3 bucket. The subsequent events include the message (a tab delimited log line) and a field called "cloudfront\_fields" that identifies each field in the log.

It is straightforward to mutate the "cloudfront\_fields" into an array of column headers:

```
{
  "type" => "cloudfront",  
  "cloudfront_version" => "1.0",
  "@version" => "1",
  "@timestamp" => 2019-01-04T22:08:40.319Z,
  "message" => "tab\tdelimited\tlog\entry\t...",
  "cloudfront_fields" => [
     [0] "date",
     [1] "time",
     [2] "x-edge-location",
     [3] "sc-bytes",
     [4] "c-ip",
     [5] "cs-method",
     [6] "cs(Host)",
     [7] "cs-uri-stem",
     [8] "sc-status",
     [9] "cs(Referer)",
     [10] "cs(User-Agent)",
     [11] "cs-uri-query",
     [12] "cs(Cookie)",
     [13] "x-edge-result-type",
     [14] "x-edge-request-id",
     [15] "x-host-header",
     [16] "cs-protocol",
     [17] "cs-bytes",
     [18] "time-taken",
     [19] "x-forwarded-for",
     [20] "ssl-protocol",
     [21] "ssl-cipher",
     [22] "x-edge-response-result-type",
     [23] "cs-protocol-version",
     [24] "fle-status",
     [25] "fle-encrypted-fields"
  ]
}

```

I cannot figure out how to use the `[cloudfront_fields]` array as the "columns" input for the CSV filter:

```
input {
  s3 {
    "type" => "cloudfront"
    "bucket" => "${S3_BUCKET}"
    "prefix" => "${S3_BUCKET_PREFIX}"
    "region" => "${S3_BUCKET_REGION}"
    "additional_settings" => {
          "force_path_style" => true
          "follow_redirects" => false
     }
  }
}

filter {
  mutate {
    split => { "cloudfront_fields" => " " }
  }
  csv {
    separator => "\t"
    columns => [cloudfront_fields] # <-- everything I try here doesn't work
    target => "csv"
  }
}

output {
  stdout { codec => "rubydebug" }
}

```

I've read through [Accessing Event Data and Fields in the Configuration](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html) and it does not appear to address this use case. (or perhaps this use case deviates from those instructions)

Yes, I could hardcode a grok pattern. But it seems more robust to use the provided event data to map the fields. This will also more-gracefully handle the addition/removal of fields if that should ever occur.

I am also open to suggestions of other filter plugin combinations that would accomplish the same thing and would still be more succinct than writing a ruby script.

Thanks.

---

<div class="post-metadata">

**Author:** ![wwalker](https://avatars.discourse-cdn.com/v4/letter/w/43a26b/32.png) [@wwalker](https://discuss.elastic.co/u/wwalker)\
**Post date:** [January 5, 2019, 10:51pm UTC](https://discuss.elastic.co/t/logstash-csv-filter-define-columns-from-event-data-fields/163014/2 "2019-01-05T22:51:42Z")

</div>

The CSV plugin assumes the header column is a single string. If you can convert the array into a single string then it should work.

---

<div class="post-metadata">

**Author:** ![dm90](https://avatars.discourse-cdn.com/v4/letter/d/e8c25b/32.png) [@dm90](https://discuss.elastic.co/u/dm90)\
**Post date:** [January 6, 2019, 4:12pm UTC](https://discuss.elastic.co/t/logstash-csv-filter-define-columns-from-event-data-fields/163014/3 "2019-01-06T16:12:45Z")

</div>

Are you referring to the CSV filter plugin? (not the input plugin)

The filter plugin requires an array: [https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html#plugins-filters-csv-columns](https://www.elastic.co/guide/en/logstash/current/plugins-filters-csv.html#plugins-filters-csv-columns)

Am I misunderstanding the doc?

(I ended up just using Ruby for this, but still interested in an answer)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 3, 2019, 4:12pm UTC](https://discuss.elastic.co/t/logstash-csv-filter-define-columns-from-event-data-fields/163014/4 "2019-02-03T16:12:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.