I have confirmed that’s the behavior as well. In event.rb:
def init_timestamp(o)
begin
timestamp = o ? LogStash::Timestamp.coerce(o) : LogStash::Timestamp.now
You can see that if the @timestamp field already exists, it uses that for the timestamp, otherwise uses the current time.
And then in elasticsearch.rb, it formats the YYYY.MM.DD with the timestamp:
index = event.sprintf(@index)
where sprintf uses the @timestamp field.
So the key is to make sure either your client sets @timestamp or you use the date field on the log stash side.
Thanks!
On September 30, 2014 at 2:55:32 PM, Brian (brian.from.fl@gmail.com) wrote:
Matt,
Assuming your logstash configurations correctly set the @timestamp field, then logstash will store the document in the day that is specified by the @timestamp field.
I have verified this behavior by observation over the time we have been using the ELK stack.
For example, we have a Perl CGI script that is used to emulate a customer service. It has a hard-coded ISO-8601 date string which our logstash configuration finds before it notices the syslog date. And so that log entry ends up in the day in the past that the hard-coded string specifies. And then curator cleans it up each and every day.
Bottom line: logstash already respects the day in the @timestamp when storing data in ES.
Brian
On Tuesday, September 30, 2014 2:31:59 PM UTC-4, Matt Hughes wrote:
I have a logstash-forwarder client sending events to lumberjack -> elasticsearch to timestamped logstash indices. How does logstash decide what day index to put the document in. Does it look at @timestamp? @timestamp is just generated when the document is received, correct? So if you logged an event on a client at 11 pm UTC but it didn't make it to elasticsearch until 1am UTC the next day, which index would it go in? Would it go in the day it was created or would it go in the day it got to elasticsearch?
If the latter, is there a way to force logstash to respect a date field in the original log event?
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/xCDVA7zTO6w/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3931b0d7-6923-4dce-a524-33b49d04af01%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.542afdad.625558ec.2008%40Matthews-MacBook-Pro.local.
For more options, visit https://groups.google.com/d/optout.