# Duplicate Entries in Filebeat Registry File For Kubernetes Pod Restart
**URL:** https://discuss.elastic.co/t/duplicate-entries-in-filebeat-registry-file-for-kubernetes-pod-restart/242564
**Category:** Beats
**Tags:** docker, filebeat
**Created:** [July 24, 2020, 7:55pm UTC](https://discuss.elastic.co/t/duplicate-entries-in-filebeat-registry-file-for-kubernetes-pod-restart/242564 "2020-07-24T19:55:59Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@pulkit007](https://discuss.elastic.co/u/pulkit007)
#### Post date: [July 24, 2020, 7:55pm UTC](https://discuss.elastic.co/t/duplicate-entries-in-filebeat-registry-file-for-kubernetes-pod-restart/242564/1 "2020-07-24T19:55:59Z")
Hi,
I am running filebeat in a Docker Container on Kubernetes Cluster for Processing Logs of our Application and send them to Logstash. Our Log Data is stored in PV, so I ran only one pod of filebeat that takes logs from that PV, process it and send them to Logstash. I am storing Filebeat Registry file in our PV so that in case of any restart, it should start from wherever it left.
Now the problem arises with Pod restarts. As the new pod may be deployed on any node in kubernetes cluster, the **device** field in Filebeat Registry file gets changed and it is not able to retrieve offset information for a log file with the new (device + inode) combination, so it creates new entry for that log file in registry file with the new (device + inode) combination.
Registry (data.json) file before pod restart :
```auto
{
"source": "my_service_.2020-06-16.27.log",
"offset": 158130,
"timestamp": "2020-07-24T18:43:28.716910538Z",
"ttl": -1,
"type": "log",
"meta": null,
"FileStateOS": {
"inode": 4187121761,
"device": 2097258
}
}
```
Registry (data.json) file after pod restart :
```auto
{
"source": "my_service_.2020-06-16.27.log",
"offset": 158130,
"timestamp": "2020-07-24T18:43:28.716910538Z",
"ttl": -1,
"type": "log",
"meta": null,
"FileStateOS": {
"inode": 4187121761,
"device": 2097258
}
},
{
"source": "my_service_.2020-06-16.27.log",
"offset": 210589,
"timestamp": "2020-07-24T18:53:59.49175077Z",
"ttl": -1,
"type": "log",
"meta": null,
"FileStateOS": {
"inode": 4187121761,
"device": 8388788
}
}
```
This duplicate entry in the Registry file causes it to reprocess that log file again.. leading to duplicate logs in Elasticsearch.
As our log data is very large.. we cannot afford to reprocess all the log files for every pod restart.
- Is there any solution to this problem??
- Is there any way in which **device** filed can be kept as same over pod restarts??
Please provide solution asap as this is a very crucial part in our deployment.
---
### Author:  [@mtojek](https://discuss.elastic.co/u/mtojek)
#### Post date: [July 27, 2020, 9:10am UTC](https://discuss.elastic.co/t/duplicate-entries-in-filebeat-registry-file-for-kubernetes-pod-restart/242564/2 "2020-07-27T09:10:45Z")
Take a look at `file_identity` described here: [https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-input-log.html](https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-input-log.html)
---
### Author:  [@pulkit007](https://discuss.elastic.co/u/pulkit007)
#### Post date: [July 27, 2020, 4:45pm UTC](https://discuss.elastic.co/t/duplicate-entries-in-filebeat-registry-file-for-kubernetes-pod-restart/242564/3 "2020-07-27T16:45:04Z")
Hi Marcin,
Thanks for the link.
I ran "lsblk -o MOUNTPOINT,UUID" in Kubernetes cluster. Output was bunch of empty lines. So, I could not continue with inode\_marker as file\_identity. Is there any other soultion for this?
Meanwhile, I tried to run filebeat in my local using path as file\_identity. After filebeat processed the log file completely, I renamed the log file to a new name. So, as per the logic, filebeat should start harvesting this file again (as the path has changed and it is using path as file\_identity), but harvesting didn’t started. I checked the registry file and it is using the inode and device for FileStateOS.
My filebeat.yml looks like this :
```auto
filebeat.inputs:
- type: log
file_identity.path: ~
recursive_glob.enabled: true
paths:
- ../Logs/**/*.log*
```
So, why the **path** as **file\_identity** is not working ?
---
### Author:  [@pulkit007](https://discuss.elastic.co/u/pulkit007)
#### Post date: [July 28, 2020, 2:01pm UTC](https://discuss.elastic.co/t/duplicate-entries-in-filebeat-registry-file-for-kubernetes-pod-restart/242564/4 "2020-07-28T14:01:17Z")
I used **path** as **file\_identity** in Kubernetes also.
I added `file_identity.path: ~ ` in filebeat log input and deployed it to Kubernetes. After some time, when the processing of all files was completed, I restarted the pod. Then filebeat started processing all the files once again. But this time registry file after pod restart does not contains values with old device id, it contains values with the new device id only.
Registry (data.json) file before pod restart :
```auto
{
"source": "my_service_.2020-05-08.12.log",
"offset": 52433912,
"timestamp": "2020-07-28T17:24:16.527054115Z",
"ttl": -1,
"type": "log",
"meta": null,
"FileStateOS": {
"inode": 4180331148,
"device": 5242882
}
}
```
Registry (data.json) file after pod restart :
```auto
{
"source": "my_service_.2020-05-08.12.log",
"offset": 94627,
"timestamp": "2020-07-28T13:42:50.376340649Z",
"ttl": -1,
"type": "log",
"meta": null,
"FileStateOS": {
"inode": 4180331148,
"device": 3145768
}
}
```
Still reprocessing of all files happened which lead to duplicated records in ES.
So, why the **path** as **file\_identity** is not working ?
---
### Author:  [@kvch](https://discuss.elastic.co/u/kvch)
#### Post date: [July 30, 2020, 3:04pm UTC](https://discuss.elastic.co/t/duplicate-entries-in-filebeat-registry-file-for-kubernetes-pod-restart/242564/5 "2020-07-30T15:04:48Z")
file\_identity is going to be released in 7.9. What version are you running?
---
### Author:  [@pulkit007](https://discuss.elastic.co/u/pulkit007)
#### Post date: [July 30, 2020, 3:23pm UTC](https://discuss.elastic.co/t/duplicate-entries-in-filebeat-registry-file-for-kubernetes-pod-restart/242564/6 "2020-07-30T15:23:07Z")
I am using Filebeat 7.8.1
Will wait for Filebeat 7.9
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 27, 2020, 5:23pm UTC](https://discuss.elastic.co/t/duplicate-entries-in-filebeat-registry-file-for-kubernetes-pod-restart/242564/7 "2020-08-27T17:23:10Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.