How does logstash chose which timestamped index to use?

I have a logstash-forwarder client sending events to lumberjack ->
elasticsearch to timestamped logstash indices. How does logstash decide
what day index to put the document in. Does it look at @timestamp?
@timestamp is just generated when the document is received, correct? So if
you logged an event on a client at 11 pm UTC but it didn't make it to
elasticsearch until 1am UTC the next day, which index would it go in?
Would it go in the day it was created or would it go in the day it got to
elasticsearch?

If the latter, is there a way to force logstash to respect a date field in
the original log event?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/58cbf668-d751-4fd5-a6d3-de9c21fe0c59%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

On Tuesday, September 30, 2014 at 20:31 CEST,
Matt Hughes hughes.matt@gmail.com wrote:

I have a logstash-forwarder client sending events to lumberjack ->
elasticsearch to timestamped logstash indices. How does logstash
decide what day index to put the document in. Does it look at
@timestamp?

Yes.

@timestamp is just generated when the document is received, correct?
So if you logged an event on a client at 11 pm UTC but it didn't make
it to elasticsearch until 1am UTC the next day, which index would it
go in? Would it go in the day it was created or would it go in the
day it got to elasticsearch?
If the latter, is there a way to force logstash to respect a date field
in the original log event?

You should use a 'date' filter to extract the date and time from a field
in the log message and populate the @timestamp field.

http://logstash.net/docs/1.4.2/filters/date

This is really more of a Logstash question, and there's a separate
group for that: logstash-users@googlegroups.com

--
Magnus Bäck | Software Engineer, Development Tools
magnus.back@sonymobile.com | Sony Mobile Communications

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20140930184741.GA21595%40seldlx20533.corpusers.net.
For more options, visit https://groups.google.com/d/optout.

Matt,

Assuming your logstash configurations correctly set the @timestamp field,
then logstash will store the document in the day that is specified by the
@timestamp field.

I have verified this behavior by observation over the time we have been
using the ELK stack.

For example, we have a Perl CGI script that is used to emulate a customer
service. It has a hard-coded ISO-8601 date string which our logstash
configuration finds before it notices the syslog date. And so that log
entry ends up in the day in the past that the hard-coded string specifies.
And then curator cleans it up each and every day.

Bottom line: logstash already respects the day in the @timestamp when
storing data in ES.

Brian

On Tuesday, September 30, 2014 2:31:59 PM UTC-4, Matt Hughes wrote:

I have a logstash-forwarder client sending events to lumberjack ->
elasticsearch to timestamped logstash indices. How does logstash decide
what day index to put the document in. Does it look at @timestamp?
@timestamp is just generated when the document is received, correct? So if
you logged an event on a client at 11 pm UTC but it didn't make it to
elasticsearch until 1am UTC the next day, which index would it go in?
Would it go in the day it was created or would it go in the day it got to
elasticsearch?

If the latter, is there a way to force logstash to respect a date field in
the original log event?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3931b0d7-6923-4dce-a524-33b49d04af01%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

I have confirmed that’s the behavior as well. In event.rb:

def init_timestamp(o)
begin
timestamp = o ? LogStash::Timestamp.coerce(o) : LogStash::Timestamp.now
You can see that if the @timestamp field already exists, it uses that for the timestamp, otherwise uses the current time.

And then in elasticsearch.rb, it formats the YYYY.MM.DD with the timestamp:

index = event.sprintf(@index)

where sprintf uses the @timestamp field.

So the key is to make sure either your client sets @timestamp or you use the date field on the log stash side.

Thanks!

On September 30, 2014 at 2:55:32 PM, Brian (brian.from.fl@gmail.com) wrote:

Matt,

Assuming your logstash configurations correctly set the @timestamp field, then logstash will store the document in the day that is specified by the @timestamp field.

I have verified this behavior by observation over the time we have been using the ELK stack.

For example, we have a Perl CGI script that is used to emulate a customer service. It has a hard-coded ISO-8601 date string which our logstash configuration finds before it notices the syslog date. And so that log entry ends up in the day in the past that the hard-coded string specifies. And then curator cleans it up each and every day.

Bottom line: logstash already respects the day in the @timestamp when storing data in ES.

Brian

On Tuesday, September 30, 2014 2:31:59 PM UTC-4, Matt Hughes wrote:

I have a logstash-forwarder client sending events to lumberjack -> elasticsearch to timestamped logstash indices. How does logstash decide what day index to put the document in. Does it look at @timestamp? @timestamp is just generated when the document is received, correct? So if you logged an event on a client at 11 pm UTC but it didn't make it to elasticsearch until 1am UTC the next day, which index would it go in? Would it go in the day it was created or would it go in the day it got to elasticsearch?

If the latter, is there a way to force logstash to respect a date field in the original log event?

You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/xCDVA7zTO6w/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3931b0d7-6923-4dce-a524-33b49d04af01%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/etPan.542afdad.625558ec.2008%40Matthews-MacBook-Pro.local.
For more options, visit https://groups.google.com/d/optout.