# How does logstash chose which timestamped index to use? **URL:** \ **Category:** Elasticsearch\ **Created:** [September 30, 2014, 6:31pm UTC](https://discuss.elastic.co/t/how-does-logstash-chose-which-timestamped-index-to-use/20015 "2014-09-30T18:31:58Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![mhughes](https://avatars.discourse-cdn.com/v4/letter/m/85e7bf/32.png) [@mhughes](https://discuss.elastic.co/u/mhughes)\ **Post date:** [September 30, 2014, 6:31pm UTC](https://discuss.elastic.co/t/how-does-logstash-chose-which-timestamped-index-to-use/20015/1 "2014-09-30T18:31:58Z")
I have a logstash-forwarder client sending events to lumberjack -\> elasticsearch to timestamped logstash indices. How does logstash decide what _day_ index to put the document in. Does it look at @timestamp? @timestamp is just generated when the document is received, correct? So if you logged an event on a client at 11 pm UTC but it didn't make it to elasticsearch until 1am UTC the next day, which index would it go in? Would it go in the day it was created or would it go in the day it got to elasticsearch? If the latter, is there a way to force logstash to respect a date field in the original log event? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/58cbf668-d751-4fd5-a6d3-de9c21fe0c59%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/58cbf668-d751-4fd5-a6d3-de9c21fe0c59%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [September 30, 2014, 6:47pm UTC](https://discuss.elastic.co/t/how-does-logstash-chose-which-timestamped-index-to-use/20015/2 "2014-09-30T18:47:41Z")
On Tuesday, September 30, 2014 at 20:31 CEST, Matt Hughes [hughes.matt@gmail.com](mailto:hughes.matt@gmail.com) wrote: > I have a logstash-forwarder client sending events to lumberjack -\> > elasticsearch to timestamped logstash indices. How does logstash > decide what _day_ index to put the document in. Does it look at > @timestamp? Yes. > @timestamp is just generated when the document is received, correct? > So if you logged an event on a client at 11 pm UTC but it didn't make > it to elasticsearch until 1am UTC the next day, which index would it > go in? Would it go in the day it was created or would it go in the > day it got to elasticsearch? > If the latter, is there a way to force logstash to respect a date field > in the original log event? You should use a 'date' filter to extract the date and time from a field in the log message and populate the @timestamp field. > **[Date filter plugin | Logstash Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html)** This is really more of a Logstash question, and there's a separate group for that: [logstash-users@googlegroups.com](mailto:logstash-users@googlegroups.com) -- Magnus Bäck | Software Engineer, Development Tools [magnus.back@sonymobile.com](mailto:magnus.back@sonymobile.com) | Sony Mobile Communications -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/20140930184741.GA21595%40seldlx20533.corpusers.net](https://groups.google.com/d/msgid/elasticsearch/20140930184741.GA21595%40seldlx20533.corpusers.net). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![brian\_yoder](https://avatars.discourse-cdn.com/v4/letter/b/f1d935/32.png) [@brian\_yoder](https://discuss.elastic.co/u/brian_yoder)\ **Post date:** [September 30, 2014, 6:55pm UTC](https://discuss.elastic.co/t/how-does-logstash-chose-which-timestamped-index-to-use/20015/3 "2014-09-30T18:55:24Z")
Matt, Assuming your logstash configurations correctly set the @timestamp field, then logstash will store the document in the day that is specified by the @timestamp field. I have verified this behavior by observation over the time we have been using the ELK stack. For example, we have a Perl CGI script that is used to emulate a customer service. It has a hard-coded ISO-8601 date string which our logstash configuration finds before it notices the syslog date. And so that log entry ends up in the day in the past that the hard-coded string specifies. And then curator cleans it up each and every day. Bottom line: logstash already respects the day in the @timestamp when storing data in ES. Brian On Tuesday, September 30, 2014 2:31:59 PM UTC-4, Matt Hughes wrote: > I have a logstash-forwarder client sending events to lumberjack -\> > elasticsearch to timestamped logstash indices. How does logstash decide > what _day_ index to put the document in. Does it look at @timestamp? > @timestamp is just generated when the document is received, correct? So if > you logged an event on a client at 11 pm UTC but it didn't make it to > elasticsearch until 1am UTC the next day, which index would it go in? > Would it go in the day it was created or would it go in the day it got to > elasticsearch? > > If the latter, is there a way to force logstash to respect a date field in > the original log event? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3931b0d7-6923-4dce-a524-33b49d04af01%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3931b0d7-6923-4dce-a524-33b49d04af01%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![mhughes](https://avatars.discourse-cdn.com/v4/letter/m/85e7bf/32.png) [@mhughes](https://discuss.elastic.co/u/mhughes)\ **Post date:** [September 30, 2014, 6:59pm UTC](https://discuss.elastic.co/t/how-does-logstash-chose-which-timestamped-index-to-use/20015/4 "2014-09-30T18:59:57Z")
I have confirmed that’s the behavior as well. In event.rb: def init\_timestamp(o) begin timestamp = o ? LogStash::Timestamp.coerce(o) : LogStash::Timestamp.now You can see that if the @timestamp field already exists, it uses that for the timestamp, otherwise uses the current time. And then in elasticsearch.rb, it formats the YYYY.MM.DD with the timestamp: ``` index = event.sprintf(@index) ``` where sprintf uses the @timestamp field. So the key is to make sure either your client sets @timestamp or you use the date field on the log stash side. Thanks! On September 30, 2014 at 2:55:32 PM, Brian ([brian.from.fl@gmail.com](mailto:brian.from.fl@gmail.com)) wrote: Matt, Assuming your logstash configurations correctly set the @timestamp field, then logstash will store the document in the day that is specified by the @timestamp field. I have verified this behavior by observation over the time we have been using the ELK stack. For example, we have a Perl CGI script that is used to emulate a customer service. It has a hard-coded ISO-8601 date string which our logstash configuration finds before it notices the syslog date. And so that log entry ends up in the day in the past that the hard-coded string specifies. And then curator cleans it up each and every day. Bottom line: logstash already respects the day in the @timestamp when storing data in ES. Brian On Tuesday, September 30, 2014 2:31:59 PM UTC-4, Matt Hughes wrote: I have a logstash-forwarder client sending events to lumberjack -\> elasticsearch to timestamped logstash indices. How does logstash decide what _day_ index to put the document in. Does it look at @timestamp? @timestamp is just generated when the document is received, correct? So if you logged an event on a client at 11 pm UTC but it didn't make it to elasticsearch until 1am UTC the next day, which index would it go in? Would it go in the day it was created or would it go in the day it got to elasticsearch? ## If the latter, is there a way to force logstash to respect a date field in the original log event? You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/xCDVA7zTO6w/unsubscribe](https://groups.google.com/d/topic/elasticsearch/xCDVA7zTO6w/unsubscribe). To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3931b0d7-6923-4dce-a524-33b49d04af01%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3931b0d7-6923-4dce-a524-33b49d04af01%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/etPan.542afdad.625558ec.2008%40Matthews-MacBook-Pro.local](https://groups.google.com/d/msgid/elasticsearch/etPan.542afdad.625558ec.2008%40Matthews-MacBook-Pro.local). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:59am UTC](https://discuss.elastic.co/t/how-does-logstash-chose-which-timestamped-index-to-use/20015/5 "2017-07-06T00:59:05Z")