# CSV filter - Create Index per log timestamp **URL:** \ **Category:** Logstash\ **Created:** [March 12, 2016, 7:46pm UTC](https://discuss.elastic.co/t/csv-filter-create-index-per-log-timestamp/44234 "2016-03-12T19:46:01Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![thunderbirdgit](https://avatars.discourse-cdn.com/v4/letter/t/ea666f/32.png) [@thunderbirdgit](https://discuss.elastic.co/u/thunderbirdgit)\ **Post date:** [March 12, 2016, 7:46pm UTC](https://discuss.elastic.co/t/csv-filter-create-index-per-log-timestamp/44234/1 "2016-03-12T19:46:01Z")
Hi I want to create ES index based on the dates matching from the logfile. I am using logstash CSV filter to process the logs. For instance, the log data appears like below ``` 2016-02-21 00:02:32.238,123.abc.com,data 2016-02-22 00:04:40.145,345.abc.com,data ``` Below is the logstash configuration file. Obviously the index will be created as testlog, however, i want the index to be created as testlog-2016.02.21 and testlog-2016.02.22, given that YYYY.mm.dd is the logstash preferred format for index dates. I have done this with grok filters, and I am trying the achieve the same with csv, but this doesn't seem to work. ``` filter { csv { columns => ["timestamp", "host", "data"] separator => "," remove_field => ["message"] } } output { elasticsearch { hosts => ["localhost:9200"] index => "testlog" } } ``` We are on Logstash 2.1.0, ES 2.1.0 and Kibana 4.3.0 version Any inputs appreciated ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 12, 2016, 8:52pm UTC](https://discuss.elastic.co/t/csv-filter-create-index-per-log-timestamp/44234/2 "2016-03-12T20:52:13Z")
Take a look at [https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-index](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-index) ---
**Author:** ![thunderbirdgit](https://avatars.discourse-cdn.com/v4/letter/t/ea666f/32.png) [@thunderbirdgit](https://discuss.elastic.co/u/thunderbirdgit)\ **Post date:** [March 12, 2016, 9:57pm UTC](https://discuss.elastic.co/t/csv-filter-create-index-per-log-timestamp/44234/3 "2016-03-12T21:57:39Z")
Thanks, after changing the output filter like below, the index is however created based on today's date. I want the index to be created based on date in log file. ``` output { elasticsearch { hosts => ["localhost:9200"] index => "testlog-%{+YYYY.MM.dd}" } } ``` Index: yellow open testlog-2016.03.12 5 1 11 0 6.8kb 6.8kb ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 12, 2016, 10:22pm UTC](https://discuss.elastic.co/t/csv-filter-create-index-per-log-timestamp/44234/4 "2016-03-12T22:22:46Z")
Then you need to add a date filter to your config 🙂 ---
**Author:** ![thunderbirdgit](https://avatars.discourse-cdn.com/v4/letter/t/ea666f/32.png) [@thunderbirdgit](https://discuss.elastic.co/u/thunderbirdgit)\ **Post date:** [March 13, 2016, 8:54pm UTC](https://discuss.elastic.co/t/csv-filter-create-index-per-log-timestamp/44234/5 "2016-03-13T20:54:28Z")
Included the date filter and tried adding a new field called logdate, which is being referenced in the index name. I am getting a "\_dateparsefailure" now. I am sure I am doing definitely wrong, ``` filter { csv { columns => ["timestamp", "host", "data"] separator => "," remove_field => ["message"] } date { match => ["@timestamp", "YYYY-MM-dd"] add_field => { "logdate" => "@timestamp" } } } output { elasticsearch { hosts => ["localhost:9200"] index => "testlog-%{logdate}"" } } ``` ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 14, 2016, 12:57am UTC](https://discuss.elastic.co/t/csv-filter-create-index-per-log-timestamp/44234/6 "2016-03-14T00:57:46Z")
> [@thunderbirdgit](#): > > 2016-02-22 00:04:40.145 That is not; > [@thunderbirdgit](#): > > "YYYY-MM-dd" You need to take the time into account as well, take a look at [http://grokdebug.herokuapp.com/patterns#](http://grokdebug.herokuapp.com/patterns#) to see if you can find anything that matches. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 14, 2016, 6:44am UTC](https://discuss.elastic.co/t/csv-filter-create-index-per-log-timestamp/44234/7 "2016-03-14T06:44:43Z")
> ``` > match => ["@timestamp", "YYYY-MM-dd"] > > ``` It's the `timestamp` field you're parsing so this should be: ``` match => ["timestamp", "YYYY-MM-dd"] ``` > ``` > add_field => { "logdate" => "@timestamp" } > > ``` Why are you duplicating the timestamp into another field? ---
**Author:** ![thunderbirdgit](https://avatars.discourse-cdn.com/v4/letter/t/ea666f/32.png) [@thunderbirdgit](https://discuss.elastic.co/u/thunderbirdgit)\ **Post date:** [March 14, 2016, 4:37pm UTC](https://discuss.elastic.co/t/csv-filter-create-index-per-log-timestamp/44234/8 "2016-03-14T16:37:04Z")
I got the same dateparsefailure with the timestamp as well, so I changed it to see if @timestamp works ``` date { match => ["timestamp", "YYYY-MM-dd"] } ``` ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [March 14, 2016, 6:31pm UTC](https://discuss.elastic.co/t/csv-filter-create-index-per-log-timestamp/44234/9 "2016-03-14T18:31:26Z")
As @warkolm said you need to adjust your date pattern so that it matches your timestamp format. According to what you wrote earlier you also have hours, minutes, seconds, and milliseconds. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 5:07am UTC](https://discuss.elastic.co/t/csv-filter-create-index-per-log-timestamp/44234/10 "2017-07-06T05:07:06Z")