# Indexes creation by date issue **URL:** \ **Category:** Logstash\ **Created:** [July 7, 2017, 8:37am UTC](https://discuss.elastic.co/t/indexes-creation-by-date-issue/92233 "2017-07-07T08:37:43Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![alonsosanchezd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alonsosanchezd/32/19846_2.png) [@alonsosanchezd](https://discuss.elastic.co/u/alonsosanchezd)\ **Post date:** [July 7, 2017, 8:37am UTC](https://discuss.elastic.co/t/indexes-creation-by-date-issue/92233/1 "2017-07-07T08:37:43Z")
Hi all! Firstly, let me thank you all for your help! I have an strange behaviour of filebeat/logstash and elasticsearch. Let me explain: I have to index log files of last month and I'm forcing logstash to replace processing timestamp by event timestamp in this way: ```auto filter { if [type] == "sas" { grok { match => { "message" => "%{IP:client} - - \[%{HTTPDATE:timestamp}\] \"POST /RTDM/rest/decisions/%{GREEDYDATA:tarificacion} %{DATA:protocol}\" %{NUMBER:code} %{NUMBER:bytes}" } } date { match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] locale => "en" } } } output { stdout { codec => json_lines } elasticsearch { hosts => [ "a.local:9200", "b.local:9200", "c.local:9200" ] sniffing => true index => "sas-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" } } ``` In addition, I've configured filebeats in order to process only log files from last month: ```auto - /opt/sas/sasconfig/Lev1/Web/WebAppServer/SASServer7_1/logs/localhost_access_log.2017-06*.txt ``` The fact is that I got an index per day (Great!), but an additional index of the current day (2017-07-07) is created with a lot of documents (looks like the sum of all the others) Is there anyway to fix this, in order to get rid of the current-date index? This is the expected behaviour? Thanks a lot in advance! ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 7, 2017, 9:42am UTC](https://discuss.elastic.co/t/indexes-creation-by-date-issue/92233/2 "2017-07-07T09:42:02Z")
No, this is not expected. Do you have the configuration snippet above in a file in /etc/logstash/conf.d? Do you have any other files in that directory? Show an example of a document that ends up in the current day index. ---
**Author:** ![alonsosanchezd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alonsosanchezd/32/19846_2.png) [@alonsosanchezd](https://discuss.elastic.co/u/alonsosanchezd)\ **Post date:** [July 7, 2017, 11:42am UTC](https://discuss.elastic.co/t/indexes-creation-by-date-issue/92233/3 "2017-07-07T11:42:45Z")
Hi Magnus! Thanks a lot for your response!!! At this moment, I have the following files in a "pipeline" directory. The logstash process is started pointing to that directory 02-beats-input.conf 11-sas.conf 30-sas-output.conf ```auto input { beats { port => 5044 ssl => false codec => plain { charset => "ISO-8859-1" } } } ``` ```auto filter { if [type] == "sas" { grok { match => { "message" => "%{IP:client} - - \[%{HTTPDATE:timestamp}\] \"POST /RTDM/rest/decisions/%{GREEDYDATA:tarificacion} %{DATA:protocol}\" %{NUMBER:code} %{NUMBER:bytes}" } } date { match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] locale => "en" } } } ``` ```auto output { stdout { codec => json_lines } elasticsearch { hosts => [ "a.local:9200", "b.local:9200", "c.local:9200" ] sniffing => true index => "sas-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" } } ``` Log events are sent from machines, via filebeat, including "sas" as document type And finally, elasticsearch creates one index per day of the last month and an additional one, for the current day. I'm sure i'm doing something wrong, but I have no idea xD ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 12, 2017, 10:37am UTC](https://discuss.elastic.co/t/indexes-creation-by-date-issue/92233/4 "2017-07-12T10:37:57Z")
Show an example of a document that ends up in the current day index. ---
**Author:** ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)\ **Post date:** [July 12, 2017, 12:51pm UTC](https://discuss.elastic.co/t/indexes-creation-by-date-issue/92233/5 "2017-07-12T12:51:26Z")
Just by way of explanation, Logstash's Elasticsearch output plugin does not actually create indices. I know. It sounds confusing, right? Logstash sends repeated batch requests of "index" requests. Each line in these bulk requests will specify the index into which is should write that event's data, in your example that will be an index named `"sas-%{+YYYY.MM.dd}"`. Logstash derives the values for `YYYY.MM.dd` directly from the `@timestamp` field of each event as it streams by. After that, it is up to Elasticsearch to handle the bulk requests. If the specified index does not exist, Elasticsearch will automatically create it (unless you've disabled that feature). So, what you've described is completely normal and expected. Because filebeat does not extrapolate or convert any data, you must override the `@timestamp` value provided at ingest time with the converted value (which you have with your `date` filter). This brings up the question, "Why am I getting that data that is going into an index with the wrong date?" The most logical explanation is that the data there is not having the `@timestamp` value successfully overridden by the `date` filter, which means that the time of ingest will be reflected in the bulk request. This will happen in the event that the `grok` filter and/or the `date` filter fail. I suggest having a look at the data in that index, which will explain what the problem is. In particular, look for a `_grokparsefailure` tag in the `tags` field. ---
**Author:** ![alonsosanchezd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alonsosanchezd/32/19846_2.png) [@alonsosanchezd](https://discuss.elastic.co/u/alonsosanchezd)\ **Post date:** [July 12, 2017, 3:28pm UTC](https://discuss.elastic.co/t/indexes-creation-by-date-issue/92233/6 "2017-07-12T15:28:39Z")
Aaron, Magnus.... Thanks a lot for your help! I'm on holydays right now but as soon as i have a computer available, i Will give that a try As I said before... I'm a bit confused... Because It looks like elk creates the indexes right, but creates an additional one with all the documents (or at least, many of them) Thanks again! ---
**Author:** ![alonsosanchezd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alonsosanchezd/32/19846_2.png) [@alonsosanchezd](https://discuss.elastic.co/u/alonsosanchezd)\ **Post date:** [August 1, 2017, 9:13am UTC](https://discuss.elastic.co/t/indexes-creation-by-date-issue/92233/7 "2017-08-01T09:13:37Z")
Finally I've found the problem: I've include an exclude regexp in filebeat, in order to exclude certain patterns. In addition, there is the one for the grok expression, but every other entry not fitting into the grok pattern, created the index for the current date. Sorry for bother you... ☹ Thanks a lot!! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 29, 2017, 9:13am UTC](https://discuss.elastic.co/t/indexes-creation-by-date-issue/92233/8 "2017-08-29T09:13:56Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 7, 2017, 10:51am UTC](https://discuss.elastic.co/t/indexes-creation-by-date-issue/92233/9 "2017-11-07T10:51:13Z")
---
**Author:** ![Krunal\_kalaria](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/krunal_kalaria/32/23862_2.png) [@Krunal\_kalaria](https://discuss.elastic.co/u/Krunal_kalaria)\ **Post date:** [November 7, 2017, 11:02am UTC](https://discuss.elastic.co/t/indexes-creation-by-date-issue/92233/10 "2017-11-07T11:02:15Z")
Hi @alonsosanchezd try this timestamp: ISO8601\_TIMEZONE CISCOTIMESTAMP it should be work. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 7, 2017, 1:03pm UTC](https://discuss.elastic.co/t/indexes-creation-by-date-issue/92233/11 "2017-11-07T13:03:48Z")