# Using part of parsed date as selected output index
**URL:** https://discuss.elastic.co/t/using-part-of-parsed-date-as-selected-output-index/370170
**Category:** Logstash
**Created:** [November 7, 2024, 3:05pm UTC](https://discuss.elastic.co/t/using-part-of-parsed-date-as-selected-output-index/370170 "2024-11-07T15:05:50Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@nilsen](https://discuss.elastic.co/u/nilsen)
#### Post date: [November 7, 2024, 3:05pm UTC](https://discuss.elastic.co/t/using-part-of-parsed-date-as-selected-output-index/370170/1 "2024-11-07T15:05:50Z")
I need to select output index based on the parsed date instead of using current date `%{+YYYY.MM.dd}`, because the log file does not roll at midnight.
I have tried to research, but only solution I find is using `ruby code`:
```auto
input {
#...
}
filter {
#...
ruby {
code => '
parsedLogTimeString = event.get("logTime")
parsedLogTimeDate = DateTime.strptime(parsedLogTimeString,"%Y-%m-%d %H:%M:%S")
event.set("[@metadata][indexDate]", parsedLogTimeDate.strftime("%Y.%m.%d"))
'
}
#...
}
output {
elasticsearch {
#...
index => "my-index-%{[@metadata][indexDate]}"
#...
}
}
```
Is this a _proper_ solution? ( _I think it works, just feels hacky._ )
Maybe there is a way I can get part (YYYY.DD.MM) of my already converted date, and use it when selecting the output index, instead of using `ruby code`?
```auto
date {
match => ["logTime", "YYYY-MMM-dd;HH:mm:ss.SSS", "ISO8601"]
timezone => "Europe/Oslo"
target => ["logTime"]
}
```
---
### Author:  [@Rios](https://discuss.elastic.co/u/Rios)
#### Post date: [November 7, 2024, 6:58pm UTC](https://discuss.elastic.co/t/using-part-of-parsed-date-as-selected-output-index/370170/2 "2024-11-07T18:58:50Z")
Can you provide a logTime sample?
---
### Author:  [@nilsen](https://discuss.elastic.co/u/nilsen)
#### Post date: [November 8, 2024, 8:44am UTC](https://discuss.elastic.co/t/using-part-of-parsed-date-as-selected-output-index/370170/3 "2024-11-08T08:44:02Z")
Not sure what you mean. An example of a logTime could be: `2024-10-28 09:49:38,007`.
I think my code/.conf works, it just _feels_ very hacky.
I was hoping there was a solution like _(pseudo)_ :
> `index => "my-index-%{logTime.sformat('"%Y.%m.%d')}"`
Reusing the date and then not needing any `ruby code`.
Happy Birthday!
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [November 8, 2024, 11:47am UTC](https://discuss.elastic.co/t/using-part-of-parsed-date-as-selected-output-index/370170/4 "2024-11-08T11:47:01Z")
> [@nilsen](#):
>
> I need to select output index based on the parsed date instead of using current date `%{+YYYY.MM.dd}`
Good, because that's exactly what logstash does by default. String interpolation for dates [is based on](https://github.com/elastic/logstash/blob/046ea1f5a8a5e26f82d1495bb2ed9a7a3fe96332/logstash-core/src/main/java/org/logstash/StringInterpolation.java#L98) the [@timestamp] field, not the current date.
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [November 8, 2024, 12:05pm UTC](https://discuss.elastic.co/t/using-part-of-parsed-date-as-selected-output-index/370170/5 "2024-11-08T12:05:25Z")
> [@nilsen](#):
>
> I think my code/.conf works, it just _feels_ very hacky
I think the alternative would be even more hacky.
The string interpolation of the time in the index option always uses the value of `@timestamp`, if `@timestamp` is the same as `logTime`, then you may use it, but it will be the time converted back into UTC, which cannot be change.
If I undertand you want to create the indices based on the local time, which is ahead of UTC.
The `ruby` filter you did you work as it will create a field with your `YYYY.MM.dd` value, but if you don't want to use a `ruby` filter you would need to manipulate the date string of the logTime field before using the `date` filter on it.
The following filters would have the same effect of your `ruby` filter.
```auto
mutate {
add_field => {
"[@metadata][logTime]" => "%{logTime}"
}
}
mutate {
gsub => ["[@metadata][logTime]","-","."]
split => {"[@metadata][logTime]" => " "}
}
mutate {
add_field => {
"[@metadata][index]" => "%{[@metadata][logTime][0]}"
}
}
```
The `[@metadata][index]` will be the value `YYYY.MM.dd` present in the logTime string.
---
### Author:  [@nilsen](https://discuss.elastic.co/u/nilsen)
#### Post date: [November 8, 2024, 12:41pm UTC](https://discuss.elastic.co/t/using-part-of-parsed-date-as-selected-output-index/370170/6 "2024-11-08T12:41:15Z")
I didn't know that `+{YYYY.MM.dd}` was based on `@timestamp` I thought it just got todays date. I see a lot of other people change the `@timestamp` to the parsed date. But read somewhere when I started learning Logstash that it was bad practice. So we have both, and change the Timestamp field to `logTime` when creating Data Views in Kibana.
> If I undertand you want to create the indices based on the local time, which is ahead of UTC.
That's not the main reason, we have a Logstash listening to a backup log server, the log files are updated every hour, and they don't daily roll at midnight. Also the backup job does not append new lines to the log files, it deletes the old file and makes a whole new file.
So let's say 00:35 the file is updated. Since it is past midnight, all the lines from 23:35-00:00 that is supposed to go into yesterdays index now is pushed into todays new index. But because our backup job _overrides_ the whole file, Logstash thinks it is a new file, and reads the whole file again. Meaning all of yesterdays data is pushed into todays the new index.
We are trying to fix that with:
- Using fingerprint filter plugin to sha245 hash the whole line, and use that as the `document_id`.
- Using the parsed date to decide the output index.
Thank you, I will stick to my solution with `ruby code`, thanks.