# Different headers - logstash csv filter

**URL:** <https://discuss.elastic.co/t/different-headers-logstash-csv-filter/37588>\
**Category:** Logstash\
**Created:** [December 18, 2015, 4:20pm UTC](https://discuss.elastic.co/t/different-headers-logstash-csv-filter/37588 "2015-12-18T16:20:23Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![rfplourenco](https://avatars.discourse-cdn.com/v4/letter/r/a698b9/32.png) [@rfplourenco](https://discuss.elastic.co/u/rfplourenco)\
**Post date:** [December 18, 2015, 4:20pm UTC](https://discuss.elastic.co/t/different-headers-logstash-csv-filter/37588/1 "2015-12-18T16:20:23Z")

</div>

Hello,

I'm having issues when feeding CSV data into elastic.  
My logstash .conf file monitors a results folder and injects new .csv data into elastic with the file input plugin.  
The issue is that the .csv file contains different columns from time to time, depending on the test being performed.

The output header can be any combination depending on the actual results returned by the URL.  
Only the first 15 lines are the same.

**standard:**  
avg\_ct,avg\_lt,avg\_rt,concurrency,fail,stdev\_rt,succ,throughput,perc\_95.0,perc\_0.0,perc\_99.9,perc\_90.0,perc\_100.0,perc\_99.0,perc\_50.0,label

**variation**  
avg\_ct,avg\_lt,avg\_rt,concurrency,fail,stdev\_rt,succ,throughput,perc\_95.0,perc\_0.0,perc\_99.9,perc\_90.0,perc\_100.0,perc\_99.0,perc\_50.0,rc\_200,label

**SSL variation**  
avg\_ct,avg\_lt,avg\_rt,concurrency,fail,stdev\_rt,succ,throughput,perc\_95.0,perc\_0.0,perc\_99.9,perc\_90.0,perc\_100.0,perc\_99.0,perc\_50.0,rc\_200,rc\_SSLHandshakeException,rc\_503,label

**SSL variation 2**  
avg\_ct,avg\_lt,avg\_rt,concurrency,fail,stdev\_rt,succ,throughput,perc\_95.0,perc\_0.0,perc\_99.9,perc\_90.0,perc\_100.0,perc\_99.0,perc\_50.0,rc\_200,rc\_SSLHandshakeException,rc\_503,rc\_ConnectException,rc\_SocketException,label

* * *

My .conf file is below:

```
input {
    file {
      path => "/var/lib/pbench-agent/user-benchmark_ose3_test_*/1/taurus.csv"
      start_position => "beginning"
     }
}

filter {
  csv {
        separator => ','

        columns => ["avg_ct","avg_lt","avg_rt","concurrency",
                    "fail","stdev_rt","succ","throughput",
                    "perc_95.0","perc_0.0","perc_99.9","perc_90.0",
                    "perc_100.0","perc_99.0","perc_50.0","rc_200",
                    "rc_503","rc_SSLHandshakeException","rc_ConnectException","rc_SocketException",
                    "label"]
 }

  de_dot {
     fields => ["perc_95.0","perc_0.0","perc_99.9","perc_90.0",
                        "perc_100.0","perc_99.0","perc_50.0"]
    }

output {
    elasticsearch {
        hosts => "gprfc076:9200"
        manage_template => false
        index => "bzt_pbench-%{+YYYY.MM.dd}"
    }
    stdout { codec => rubydebug }
}

}

```

* * *

How could I create different types and send them to elastic according to the csv header line?  
Wouldn't this mean changing the doc\_type/mapping/.conf file (columns) and eventually restart logstash?

Any suggestions on how to handle this would be greatly appreciated!

Thank you,

Ricardo

---

<div class="post-metadata">

**Author:** ![deepu.sundar](https://avatars.discourse-cdn.com/v4/letter/d/e0b2c6/32.png) [@deepu.sundar](https://discuss.elastic.co/u/deepu.sundar)\
**Post date:** [November 16, 2016, 3:38pm UTC](https://discuss.elastic.co/t/different-headers-logstash-csv-filter/37588/3 "2016-11-16T15:38:48Z")

</div>

Hi,

Were you able to figure out a way to resolve this issue? I have had the same problem with my indexing problem. Please share some details if you can, would appreciate your help.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:30am UTC](https://discuss.elastic.co/t/different-headers-logstash-csv-filter/37588/4 "2017-07-06T04:30:30Z")

</div>