Generate dynamic date for ES Index

Elastic Stack Logstash
1

Hi
Below is the simple ES output which indexes to e2e-YYYY.MM.dd index in ES. i.e., of the Today's date.
output {
elasticsearch {
hosts => "host:9200"
index => "e2e-%{+YYYY.MM.dd}"
document_type => "log"
}
}

However I want the Logstash to generate the date or yesterday's date (for example trunc(sysdate)-1) and specify the ES Index accordingly.

And similarly would like to search for day-1 timestamped log files in the file input.

How Can I generate the same.

2

You could generate a field (under @metadata so it isn't included in the document sent to ES) with whatever date you like and refer to that in the index setting. But why? What problem are you trying to solve by doing this?

3

Hi OK I would have to little bit research on how to do it with @metadata.

My Use case is - I would require to ship all the day wise logs of various applications to ES and planning to have day wise Index. So for instance e2e-28-11-2015 index would have all the logs of the transactions of 28th Nov 15 and so on.

The problem statement is for one of the applications I would need to collect the data/logs from the mirror database the next day i.e., on 29th Nov 15 the logstash should run to fetch 28th Nov 15 transactions from that application, hence I would need these data to be indexed to 28-11-2015. Therefore need the logic in logstash to get the "SYSDATE-1' to construct the index name.

4

Just use the date filter to parse the timestamps from the logs and populate the @timestamp field with the actual time each event occurred. Then logs from Nov 28 will end up in e2e-2015.11.28 regardless of when you let Logstash process the logs (but keep in mind that @timestamp is UTC so depending on your timezone you might see messages from Nov 28 in your local timezone end up in either e2e-2015.11.27 or e2e-2015.11.29.

5

Thanks Magnus. I thought of the same as well to parse one of the event date field and add to metadata.