# Csv and date filter **URL:** \ **Category:** Logstash\ **Created:** [July 22, 2015, 3:47am UTC](https://discuss.elastic.co/t/csv-and-date-filter/26033 "2015-07-22T03:47:05Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![zpp](https://avatars.discourse-cdn.com/v4/letter/z/54ee81/32.png) [@zpp](https://discuss.elastic.co/u/zpp)\ **Post date:** [July 22, 2015, 3:47am UTC](https://discuss.elastic.co/t/csv-and-date-filter/26033/1 "2015-07-22T03:47:05Z")
Hi, i have a logstash agent setup on a window box with below configuration. input { file { type =\> "autolog" path =\> "E:/elkcluster/log/_.log" } file { type =\> "testscv" path =\> "e:/elkcluster/test/_.\*" } } filter { if( [type] == "testscv") { csv { columns =\> ["id", "name", "job", "location", "joindate", "dummy"] separator =\> "," } } } output { redis { host =\> "xxx.xxx.xxx.xxx" data\_type =\> "list" key =\> "logstash" } } here is the sample csv file, and it worked with all fields parsed nicely. 1,aaaa,eng,l2,Oct 11 2010,1 2,bbb,mgr,l5,Oct 28 2010,1 3,ccc,mgr,l2,Nov 12 2013,1 4,ddd,eng,l4,Nov 10 2014,1 5,eee,eng,l2,Nov 18 2010,1 6,fff,eng,l2,Dec 12 2009,1 then i wanted to set the timestamp to the joindate, i added in the date filter below, but i got exception in elasticsearch. what should be the correct date format in this case? thank you. filter { if( [type] == "testscv") { csv { columns =\> ["id", "name", "job", "location", "joindate", "dummy"] separator =\> "," } date { match =\> ["joindate", "MMM dd YYYY"] } } } [2015-07-22 12:00:46,456][DEBUG][action.bulk] [logstash-2009.12.11][2] failed to execute bulk item (index) index {[logstash-2009.12.11][testscv][AU6z67dysgFpH4c5oQD7], source[{"message":["6,fff,eng,l2,Dec 12 2009,1\r"],"@version":"1","@timestamp":"2009-12-11T16:00:00.000Z","host":"xxx","path":"e:/elkcluster/test/empdata9.csv","type":"testscv","id":"6","name":"fff","job":"eng","location":"l2","joindate":"Dec 12 2009","dummy":"1"}]} org.elasticsearch.index.mapper.MapperParsingException: failed to parse [joindate] ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 22, 2015, 8:40am UTC](https://discuss.elastic.co/t/csv-and-date-filter/26033/2 "2015-07-22T08:40:49Z")
Why not delete the `joindate` field since you're parsing the string and storing it in `@timestamp` anyway? ``` date { match => ["joindate", "MMM dd YYYY"] remove_field => ["joindate"] } ``` I'd also delete the `message` field. ---
**Author:** ![zpp](https://avatars.discourse-cdn.com/v4/letter/z/54ee81/32.png) [@zpp](https://discuss.elastic.co/u/zpp)\ **Post date:** [July 22, 2015, 8:52am UTC](https://discuss.elastic.co/t/csv-and-date-filter/26033/3 "2015-07-22T08:52:09Z")
Great, it works:smile:, thank you. But why do we need to remove this field to make it work? I actually tried another option by changing the format to "YYYY-MM-dd HH:mm:ss", and manipulated the joindate in sample data to have time, and it worked as well, without configuring the remove\_field. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 22, 2015, 8:59am UTC](https://discuss.elastic.co/t/csv-and-date-filter/26033/4 "2015-07-22T08:59:18Z")
> [@zpp](#): > > But why do we need to remove this field to make it work? I'm not sure why ES attempts to parse _that_ field as a date, but obviously if we remove the field that ES chokes on things work better. > I actually tried another option by changing the format to "YYYY-MM-dd HH:mm:ss", and manipulated the joindate in sample data to have time, and it worked as well, without configuring the remove\_field. Yes, but because of the reasons stated above that shouldn't be surprising. Now you'll have two timestamp fields; `@timestamp` and `joindate`. If you're not using the date filter anymore to parse `joindate` and store it in `@timestamp` then `@timestamp` will instead be the time you ingested the data. That may or may not make sense, depending on how you're going to use the data. ---
**Author:** ![zpp](https://avatars.discourse-cdn.com/v4/letter/z/54ee81/32.png) [@zpp](https://discuss.elastic.co/u/zpp)\ **Post date:** [July 22, 2015, 11:30am UTC](https://discuss.elastic.co/t/csv-and-date-filter/26033/5 "2015-07-22T11:30:01Z")
thank you very much for the explanation ! ---
**Author:** ![setlem](https://avatars.discourse-cdn.com/v4/letter/s/ea5d25/32.png) [@setlem](https://discuss.elastic.co/u/setlem)\ **Post date:** [September 28, 2016, 6:58am UTC](https://discuss.elastic.co/t/csv-and-date-filter/26033/6 "2016-09-28T06:58:32Z")
hi magnus, I am new to ELk.i have used the same date filter but index file is not generating in elastic search and am unable to see data in kibana. ![](https://us1.discourse-cdn.com/elastic/original/2X/8/826988c71a0f7561d1f486edbcd5dba6592b66f3.PNG) here is my config file ![](https://us1.discourse-cdn.com/elastic/original/2X/4/4b54e6de40cffa5e9dafdda9e08e208f205e449a.PNG) plz help me out. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 28, 2016, 7:00am UTC](https://discuss.elastic.co/t/csv-and-date-filter/26033/7 "2016-09-28T07:00:56Z")
@setlem, please start a new thread for your question. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:36am UTC](https://discuss.elastic.co/t/csv-and-date-filter/26033/8 "2017-07-06T04:36:36Z")