Force filebeat resend ONE particular file

Elastic Stack Beats
1

How should I force a filebeat to resend just only one particular file.
Or how may I drop for only one certain file its offset to force filebeat start processing this one file again?

Again, I'm talknig only about one concrete file. Keep in mind, that I have 100 log files in one directory and pointing to same destination. But I want to restart from scratch only 1 log file and 99 files leave at theirs offsets.

Note: this 1 file first deleted, then filled up and thats why I want to resend it.

PS: maybe its possible to define in filebeat.yml proprs for it separately to force filebeat always start from scratch? And in this case is it possible to change for only this setting scan_freq to 1s, leaving all other settings with scan_freq=10s?

2

any hints or updates is it even possible?

3

Is commuity or developers even see questions like this?

4

Hi @notricky !

I think you can try to delete the entry for this file from registry file and restart Filebeat. In this the offset for this file will be recreated.

5

Thanks!
What is the best way to do so?
Isnt there other places where info is stored?

6

Any updates or ideas concerning my last question?

7

The registry is where filebeat tracks what lines on what files have been read. You should be able to remove the info for the one file ur interested in and restart filebeat as mentioned in the above post.

8

So what is the way to do that securely with no impact on consistency to the rest files and the filebeat itself?

In example, if I change registry file anyway - then the reg file is recreteated from the top or even fb is not starting anymore after that.

So what is the way to do that?

9

You could just copy the log file to a new file then it will get loaded when Filebeat starts...

Then as your original file starts to get more logs they will get picked up.

Then you can get rid of the copy...

10

Let me explain again.

I have these logs:

  • ABC.log
  • DEF.log
  • XYZ.log

ABC.log file was already processed and logs were shipped to outer system. This log is no longer being updated.
I want ABC.log to be reshipped again, while other files (DEF, XYZ) remains processed and shouldn't be reshipped.
All log files also must be kept (removes or deletings are prohibited).

So what I can do in that case?

11

Then the only way I know of is hand edit the registry and remove the entry for that log file.

Or make a copy to somewhere else and load it.

Perhaps someone else will have a better idea

12

If I do as you suggest - just double the wanted file and then delete it - what is the way to find if a file was shipped? What is the programmic command to ask filebeat if it has finished shipping of ABC.log file?

13

As I've mentioned - any edit of registry file leads to unpredictable fb behaviour. Seems like there are other places where fb stores some data to check registry's file integrity.

14

Depending on the size ...

I count the number of logs in the file.

wc -l ABC.log

Then load it and use Kibana Discover

Filter on log.file.path for that file and see if the counts match.

15

I see.
What if I'm not using elastic or kibana?

Is there any CLI interface to filebeat itself of any bash script that is applicable (the one that can be automated at last) in this case?

16

Hmmm....

Count the entries in your destination?

Otherwise not easily, Filebeat does report the number of events shipped... But it is a little hard to decipher... every 30s it reports / logs statistics which include the number events shipped you would need to capture all those and add them up...

You would need to configure for just that file so you don't accidentally load other logs.

17

  1. Keeping in mind that filebeat hashes the filename to track if the file was renamed, isnt there a possibility to add this kind of functionality as a CLI command?
    i.e.:
    $ filebeat ship <filename>
    or
    $ filebeat reset stats <filename>

  2. Also I didn't undestand registry's file contents. Is there any docs about it, 'cause there are plenty of differently formatted lines in it that provides info about log file offset, but why format differs (json-keys are sorted differently from line to line) - I didnt get it.

  3. Another history - is that occasionaly another file with name 789543.log is created right next to registry file. This file is not permanent and the number changing all the time. Didnt get it what it means as well. Any directions?

18

@notricky

Feel free to file a feature request against the beats repo here

The registry today does not have an API and the contents is not officially documented and is really not intended to be edited, edit at your your own risk.

19

Ok, thanks.

Did as you suggest and have created an issue:

Whoever is interested in this feature please follow there and contribute with thumbsup! )

2 Likes
20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.