# Logstash 7.0.1 - CSV upload with subheaders **URL:** \ **Category:** Logstash\ **Created:** [July 1, 2019, 7:54pm UTC](https://discuss.elastic.co/t/logstash-7-0-1-csv-upload-with-subheaders/188399 "2019-07-01T19:54:33Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![etsai7](https://avatars.discourse-cdn.com/v4/letter/e/ba8739/32.png) [@etsai7](https://discuss.elastic.co/u/etsai7)\ **Post date:** [July 1, 2019, 7:54pm UTC](https://discuss.elastic.co/t/logstash-7-0-1-csv-upload-with-subheaders/188399/1 "2019-07-01T19:54:33Z")
Hello, I am trying to upload a csv file using Logstash 7.0.1 that contains subheaders. For example: ```auto "netflow": { "flow_seq_num": 55729, "flowset_id": 259, "flow_end_msec": 1562002327000, "version": 9, "layer2OctetDeltaCount": 802218000, "in_bytes": 792669846, "in_pkts": 530453, "flow_start_msec": 1562002315000 } ``` However, it seems to end up all being stored in a `message:` block when uploading. The following are the csv fields for download: ```auto output{ csv{ fields => ["[node][hostname]","[node][ipaddr]", "[event][host]", "[event][type]", "@version", "@timestamp","[flow][ip_protocol]", "[flow][packets]", "[flow][src_addr]", "[flow][src_hostname]", "[flow][src_port_name]", "[flow][direction]", "[flow][traffic_locality]", "[flow][src_autonomous_system]", "[flow][dst_port_name]","[flow][input_snmp]", "[flow][ip_version]", "[flow][bytes]", "[flow][dst_addr]", "[flow][sampling_interval]", "[flow][dst_hostname]", "[flow][dst_autonomous_system]", "[flow][input_ifname]", "[netflow][flow_seq_num]", "[netflow][flowset_id]","[netflow][flow_end_msec]", "[netflow][version]", "[netflow][layer2OctetDeltaCount]","[netflow][in_bytes]","[netflow][in_pkts]","[netflow][flow_start_msec]"] path => "Flow_csv.csv" } } ``` For CSV Upload, I use the same fields, but for the columns: ```auto input { file { path => "home/data/*.csv" start_position => "beginning" sincedb_path => "/dev/null" } } filter { csv { separator => "," columns => ["[node][hostname]","[node][ipaddr]", "[event][host]", "[event][type]", "@version", "@timestamp","[flow][ip_protocol]", "[flow][packets]", "[flow][src_addr]", "[flow][src_hostname]", "[flow][src_port_name]", "[flow][direction]", "[flow][traffic_locality]", "[flow][src_autonomous_system]", "[flow][dst_port_name]","[flow][input_snmp]", "[flow][ip_version]", "[flow][bytes]", "[flow][dst_addr]", "[flow][sampling_interval]", "[flow][dst_hostname]", "[flow][dst_autonomous_system]", "[flow][input_ifname]", "[netflow][flow_seq_num]", "[netflow][flowset_id]","[netflow][flow_end_msec]", "[netflow][version]", "[netflow][layer2OctetDeltaCount]","[netflow][in_bytes]","[netflow][in_pkts]","[netflow][flow_start_msec]"] } } output { elasticsearch { hosts => "ipaddress.com:9200" index => "elastiflow-2019-07-01" } } ``` The errors I get: ```auto [2019-07-01T15:31:36,841][WARN][logstash.filters.csv] Error parsing csv {:field=>"message", :source=>"10.1.1.253,10.1.1.253,10.1.1.253,netflow,3.5.0,2019-07-01T19:05:19.000Z,UDP,3,10.1.1.253,10.1.1.253,UDP/50101,unspecified,private,private,netflow (UDP/2055),541,IPv4,400,10.2.1.50,0,10.2.1.50,private,index: 541,149803,257,,9,,400,3,", :exception=>#} ``` Is there an issue with the upload formatting? Thanks, Eric ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 1, 2019, 8:09pm UTC](https://discuss.elastic.co/t/logstash-7-0-1-csv-upload-with-subheaders/188399/2 "2019-07-01T20:09:08Z")
The field @timestamp has type Logstash::TimeStamp, but you are trying to set it to a string value. In your list of columns, change "@timestamp" to "[@metadata][timestamp]" then add a date filter to parse it ``` date { match => ["[@metadata][timestamp]", ISO8601 ] } ``` ---
**Author:** ![etsai7](https://avatars.discourse-cdn.com/v4/letter/e/ba8739/32.png) [@etsai7](https://discuss.elastic.co/u/etsai7)\ **Post date:** [July 1, 2019, 9:37pm UTC](https://discuss.elastic.co/t/logstash-7-0-1-csv-upload-with-subheaders/188399/3 "2019-07-01T21:37:31Z")
That worked! Thank you Eric ---
**Author:** ![etsai7](https://avatars.discourse-cdn.com/v4/letter/e/ba8739/32.png) [@etsai7](https://discuss.elastic.co/u/etsai7)\ **Post date:** [July 1, 2019, 10:03pm UTC](https://discuss.elastic.co/t/logstash-7-0-1-csv-upload-with-subheaders/188399/4 "2019-07-01T22:03:21Z")
One more question, when I uploaded the editted conf file, is there a way to let ES create the types associated with each field? Or is that something that will manually have to be done in the management of Kibana? Thanks, Eric ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [July 1, 2019, 10:07pm UTC](https://discuss.elastic.co/t/logstash-7-0-1-csv-upload-with-subheaders/188399/5 "2019-07-01T22:07:31Z")
Yes, you can iterate over the fields in an object and see which can be converted to numeric forms using a ruby filter with code like [this](https://discuss.elastic.co/t/convert-values-in-kv-pairs-using-ruby-filter/135035/2). ---
**Author:** ![etsai7](https://avatars.discourse-cdn.com/v4/letter/e/ba8739/32.png) [@etsai7](https://discuss.elastic.co/u/etsai7)\ **Post date:** [July 2, 2019, 11:17pm UTC](https://discuss.elastic.co/t/logstash-7-0-1-csv-upload-with-subheaders/188399/6 "2019-07-02T23:17:34Z")
Great, Thank you! Eric ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 30, 2019, 11:17pm UTC](https://discuss.elastic.co/t/logstash-7-0-1-csv-upload-with-subheaders/188399/7 "2019-07-30T23:17:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.