Need inode or file timestamp in the event

Elastic Stack Beats
1

Hi,

We are facing a log rotation problem while filebeat is streaming a particular file. I have a File named Log_34.txt and when it was streaming it gets renamed to Log_34.txt2018 and a new file with Log_34.txt gets created. Now at the logstash we are receiving lines from both the files with the "source" (filename) as same(Log_34.txt). So we are unable to distinguish whose lines are those. Can we add a Unique ID something like an Inode or a filetimestamp in the event info while filebeat streams. Or can i get any other solution for this problem ?

Thanks

2

Hello @laxman1 inside Filebeat we do collect the inode of the file and the volume id but we do not propagate that in the event send to Logstash.

Why do you need that distinction between the files?

3

Hi @pierhugues ,

I am receiving lines from both the logs with the same filename. In my case the log content from the rotated log and the new log would of different jobs . And logstash events in my case would be processed by different application. So i couldnt associate the content to a Job in my application.

If there is a Inode or someother unique Id for a file created and filebeat propagates it I can distinguish in my application.

4

like I've said, we do not send the inode as part of the event.

You said it was two jobs, it might be simplier to use different name per job invocation?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.