How to create index as per file creation date?

Elastic Stack Logstash
1

Hi Experts ,

For now I am creating index in ES on daily basis with below logic

output {

            elasticsearch {
                    action => "index"
                    hosts => ["elastic:1234"]
                    index => "test-%{+YYYY.MM.dd}"
                    document_type => "abc"
            }

}

Unfortunately from 30th sep my LS service was down and below files are still waiting to get indexed . If I start LS today (03-10-2019)it will parse and save data in today's Index which I do not want . I want to create index as per file date . I have files like below
30_09_2019_test.log
01_10_2019_test.log
02_10_2019_test.log
03_10_2019_test.log

And I want to create index as below
test-2019.09.30
test-2019.10.01
test-2019.10.02

I hope I was clear with what I want to achieve . Please let me know if you need further info which I can provide .

Regards
VG

2

"test-%{+YYYY.MM.dd}" is evaluated using the [@timestamp] field. You need to use a date filter to parse a field that contains the date of the events to set [@timestamp].

1 Like
3

Well this is exactly what I was thinking but here is the challenge that I have the date filter that I have rt now is on another field and I am not treating this as timestamp field . Here is what I have

date {match => ["managerReceiptTime","dd MMM YYYY HH:mm:ss 'IST'"] target => "managerReceiptTime"}

and at kibana level I am defining index pattern based on this date . So is there any way that "test-%{+YYYY.MM.dd}" settings picks date from "managerReceiptTime" or from file name than timestamp field . If not then I have to convert managerReceiptTime and target to timestamp which I do not want as things will be disturbed . Can you suggest any other workaround ?

VG

4

You could build the index name in a ruby filter.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.