# Force filebeat resend ONE particular file

**URL:** https://discuss.elastic.co/t/force-filebeat-resend-one-particular-file/267115
**Category:** Beats
**Tags:** filebeat
**Created:** [March 12, 2021, 5:00pm UTC](https://discuss.elastic.co/t/force-filebeat-resend-one-particular-file/267115 "2021-03-12T17:00:16Z")
**Posts on this page:** 20
**Page:** 1

<div class="post-metadata">

### Author: ![notricky](https://avatars.discourse-cdn.com/v4/letter/n/bbce88/32.png) [@notricky](https://discuss.elastic.co/u/notricky)
#### Post date: [March 12, 2021, 5:00pm UTC](https://discuss.elastic.co/t/force-filebeat-resend-one-particular-file/267115/1 "2021-03-12T17:00:16Z")

</div>

How should I force a filebeat to resend just only **one particular file**.  
Or how may I drop for only one certain file its offset to force filebeat start processing this **one file** again?

Again, I'm talknig only about one concrete file. Keep in mind, that I have 100 log files in one directory and pointing to same destination. But I want to restart from scratch only 1 log file and 99 files leave at theirs offsets.

Note: this 1 file first deleted, then filled up and thats why I want to resend it.

PS: maybe its possible to define in filebeat.yml proprs for it separately to force filebeat always start from scratch? And in this case is it possible to change for only this setting scan\_freq to 1s, leaving all other settings with scan\_freq=10s?

---

<div class="post-metadata">

### Author: ![notricky](https://avatars.discourse-cdn.com/v4/letter/n/bbce88/32.png) [@notricky](https://discuss.elastic.co/u/notricky)
#### Post date: [March 15, 2021, 11:25pm UTC](https://discuss.elastic.co/t/force-filebeat-resend-one-particular-file/267115/2 "2021-03-15T23:25:31Z")

</div>

any hints or updates is it even possible?

---

<div class="post-metadata">

### Author: ![notricky](https://avatars.discourse-cdn.com/v4/letter/n/bbce88/32.png) [@notricky](https://discuss.elastic.co/u/notricky)
#### Post date: [March 20, 2021, 11:28pm UTC](https://discuss.elastic.co/t/force-filebeat-resend-one-particular-file/267115/3 "2021-03-20T23:28:16Z")

</div>

Is commuity or developers even see questions like this?

---

<div class="post-metadata">

### Author: ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)
#### Post date: [March 22, 2021, 8:55am UTC](https://discuss.elastic.co/t/force-filebeat-resend-one-particular-file/267115/4 "2021-03-22T08:55:01Z")

</div>

Hi @notricky !

I think you can try to delete the entry for this file from [registry file](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-general-options.html#_registry_path) and restart Filebeat. In this the offset for this file will be recreated.

---

<div class="post-metadata">

### Author: ![notricky](https://avatars.discourse-cdn.com/v4/letter/n/bbce88/32.png) [@notricky](https://discuss.elastic.co/u/notricky)
#### Post date: [March 28, 2021, 6:50pm UTC](https://discuss.elastic.co/t/force-filebeat-resend-one-particular-file/267115/5 "2021-03-28T18:50:58Z")

</div>

Thanks!  
What is the best way to do so?  
Isnt there other places where info is stored?

---

<div class="post-metadata">

### Author: ![notricky](https://avatars.discourse-cdn.com/v4/letter/n/bbce88/32.png) [@notricky](https://discuss.elastic.co/u/notricky)
#### Post date: [April 11, 2021, 11:44pm UTC](https://discuss.elastic.co/t/force-filebeat-resend-one-particular-file/267115/6 "2021-04-11T23:44:48Z")

</div>

Any updates or ideas concerning my last question?

---

<div class="post-metadata">

### Author: ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)
#### Post date: [April 12, 2021, 12:19am UTC](https://discuss.elastic.co/t/force-filebeat-resend-one-particular-file/267115/7 "2021-04-12T00:19:21Z")

</div>

The registry is where filebeat tracks what lines on what files have been read. You should be able to remove the info for the one file ur interested in and restart filebeat as mentioned in the above post.

---

<div class="post-metadata">

### Author: ![notricky](https://avatars.discourse-cdn.com/v4/letter/n/bbce88/32.png) [@notricky](https://discuss.elastic.co/u/notricky)
#### Post date: [April 16, 2021, 11:40pm UTC](https://discuss.elastic.co/t/force-filebeat-resend-one-particular-file/267115/8 "2021-04-16T23:40:27Z")

</div>

So what is the way to do that securely with no impact on consistency to the rest files and the filebeat itself?

In example, if I change registry file anyway - then the reg file is recreteated from the top or even fb is not starting anymore after that.

So what is the way to do that?

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [April 17, 2021, 12:03am UTC](https://discuss.elastic.co/t/force-filebeat-resend-one-particular-file/267115/9 "2021-04-17T00:03:11Z")

</div>

You could just copy the log file to a new file then it will get loaded when Filebeat starts...

Then as your original file starts to get more logs they will get picked up.

Then you can get rid of the copy...

---

<div class="post-metadata">

### Author: ![notricky](https://avatars.discourse-cdn.com/v4/letter/n/bbce88/32.png) [@notricky](https://discuss.elastic.co/u/notricky)
#### Post date: [April 17, 2021, 12:16am UTC](https://discuss.elastic.co/t/force-filebeat-resend-one-particular-file/267115/10 "2021-04-17T00:16:43Z")

</div>

Let me explain again.

I have these logs:

- ABC.log
- DEF.log
- XYZ.log

**ABC.log** file was already processed and logs were shipped to outer system. This log is no longer being updated.  
I want **ABC.log** to be reshipped again, while other files (DEF, XYZ) remains processed and shouldn't be reshipped.  
All log files also must be kept (removes or deletings are prohibited).

So what I can do in that case?

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [April 17, 2021, 12:23am UTC](https://discuss.elastic.co/t/force-filebeat-resend-one-particular-file/267115/11 "2021-04-17T00:23:13Z")

</div>

Then the only way I know of is hand edit the registry and remove the entry for that log file.

Or make a copy to somewhere else and load it.

Perhaps someone else will have a better idea

---

<div class="post-metadata">

### Author: ![notricky](https://avatars.discourse-cdn.com/v4/letter/n/bbce88/32.png) [@notricky](https://discuss.elastic.co/u/notricky)
#### Post date: [April 17, 2021, 12:24am UTC](https://discuss.elastic.co/t/force-filebeat-resend-one-particular-file/267115/12 "2021-04-17T00:24:44Z")

</div>

If I do as you suggest - just double the wanted file and then delete it - what is the way to find if a file was shipped? What is the programmic command to ask filebeat if it has finished shipping of **ABC.log** file?

---

<div class="post-metadata">

### Author: ![notricky](https://avatars.discourse-cdn.com/v4/letter/n/bbce88/32.png) [@notricky](https://discuss.elastic.co/u/notricky)
#### Post date: [April 17, 2021, 12:26am UTC](https://discuss.elastic.co/t/force-filebeat-resend-one-particular-file/267115/13 "2021-04-17T00:26:08Z")

</div>

As I've mentioned - any edit of registry file leads to unpredictable fb behaviour. Seems like there are other places where fb stores some data to check registry's file integrity.

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [April 17, 2021, 12:32am UTC](https://discuss.elastic.co/t/force-filebeat-resend-one-particular-file/267115/14 "2021-04-17T00:32:35Z")

</div>

Depending on the size ...

I count the number of logs in the file.

`wc -l ABC.log`

Then load it and use Kibana Discover

Filter on `log.file.path` for that file and see if the counts match.

---

<div class="post-metadata">

### Author: ![notricky](https://avatars.discourse-cdn.com/v4/letter/n/bbce88/32.png) [@notricky](https://discuss.elastic.co/u/notricky)
#### Post date: [April 17, 2021, 12:34am UTC](https://discuss.elastic.co/t/force-filebeat-resend-one-particular-file/267115/15 "2021-04-17T00:34:57Z")

</div>

I see.  
What if I'm not using elastic or kibana?

Is there any CLI interface to filebeat itself of any bash script that is applicable (the one that can be automated at last) in this case?

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [April 17, 2021, 12:41am UTC](https://discuss.elastic.co/t/force-filebeat-resend-one-particular-file/267115/16 "2021-04-17T00:41:43Z")

</div>

Hmmm....

Count the entries in your destination?

Otherwise not easily, Filebeat does report the number of events shipped... But it is a little hard to decipher... every 30s it reports / logs statistics which include the number events shipped you would need to capture all those and add them up...

You would need to configure for just that file so you don't accidentally load other logs.

---

<div class="post-metadata">

### Author: ![notricky](https://avatars.discourse-cdn.com/v4/letter/n/bbce88/32.png) [@notricky](https://discuss.elastic.co/u/notricky)
#### Post date: [April 17, 2021, 2:16am UTC](https://discuss.elastic.co/t/force-filebeat-resend-one-particular-file/267115/17 "2021-04-17T02:16:41Z")

</div>

1. Keeping in mind that filebeat hashes the filename to track if the file was renamed, isnt there a possibility to add this kind of functionality as a CLI command?  
i.e.:  
`$ filebeat ship <filename>`  
or  
`$ filebeat reset stats <filename>`

2. Also I didn't undestand registry's file contents. Is there any docs about it, 'cause there are plenty of differently formatted lines in it that provides info about log file offset, but why format differs (json-keys are sorted differently from line to line) - I didnt get it.

3. Another history - is that occasionaly another file with name 789543.log is created right next to registry file. This file is not permanent and the number changing all the time. Didnt get it what it means as well. Any directions?

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [April 17, 2021, 2:28am UTC](https://discuss.elastic.co/t/force-filebeat-resend-one-particular-file/267115/18 "2021-04-17T02:28:08Z")

</div>

@notricky

Feel free to file a feature request against the beats repo [here](https://github.com/elastic/beats/issues)

The registry today does not have an API and the contents is not officially documented and is really not intended to be edited, edit at your your own risk.

---

<div class="post-metadata">

### Author: ![notricky](https://avatars.discourse-cdn.com/v4/letter/n/bbce88/32.png) [@notricky](https://discuss.elastic.co/u/notricky)
#### Post date: [April 20, 2021, 10:11am UTC](https://discuss.elastic.co/t/force-filebeat-resend-one-particular-file/267115/19 "2021-04-20T10:11:09Z")

</div>

Ok, thanks.

Did as you suggest and have created an issue:

> <https://github.com/elastic/beats/issues/25170>
>
> A CLI command interface to filebeat to re-ship certain log file
> I want to re-ship a certain log file (or files by...

Whoever is interested in this feature please follow there and contribute with thumbsup! )

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 18, 2021, 12:11pm UTC](https://discuss.elastic.co/t/force-filebeat-resend-one-particular-file/267115/20 "2021-05-18T12:11:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
