# Can we get FileBeats to optionally ignore DeviceID in FileStateOS?

**URL:** <https://discuss.elastic.co/t/can-we-get-filebeats-to-optionally-ignore-deviceid-in-filestateos/105967>\
**Category:** Beats\
**Created:** [November 1, 2017, 12:43am UTC](https://discuss.elastic.co/t/can-we-get-filebeats-to-optionally-ignore-deviceid-in-filestateos/105967 "2017-11-01T00:43:11Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![xrl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xrl/32/3556_2.png) [@xrl](https://discuss.elastic.co/u/xrl)\
**Post date:** [November 1, 2017, 12:43am UTC](https://discuss.elastic.co/t/can-we-get-filebeats-to-optionally-ignore-deviceid-in-filestateos/105967/1 "2017-11-01T00:43:11Z")

</div>

I am running a data pipeline through filebeats, running on kubernetes. One container in the kubernetes pod receives traffic and dumps it to disk, the other container is filebeat. All traffic comes in to the pod through ELB, that way we continue processing data if a kube-node disappears. This pod has an overflow drives using a volume mount in kubernetes -- one ELB is re-used (so the inodes are consistent) but everytime it moves kube-nodes it gets a new DeviceID. This is causing files to be reprocessed during normal pod updates.

What I'm proposing is that we change the `IsSame` function ([https://github.com/elastic/beats/blob/edde8912793f3055f08937fb88fe432e47a8baaa/filebeat/input/file/file\_other.go#L29-L32](https://github.com/elastic/beats/blob/edde8912793f3055f08937fb88fe432e47a8baaa/filebeat/input/file/file_other.go#L29-L32)) to optionally ignore the DeviceID? Something like:

> // IsSame file checks if the files are identical  
> func (fs StateOS) IsSame(state StateOS, ignoreDevice bool) bool {  
> return fs.Inode == state.Inode && (ignoreDevice || (fs.Device == state.Device))  
> }

Thoughts?

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [November 1, 2017, 12:54pm UTC](https://discuss.elastic.co/t/can-we-get-filebeats-to-optionally-ignore-deviceid-in-filestateos/105967/2 "2017-11-01T12:54:53Z")

</div>

Filebeat must be able to detect file renames/moves. This requires the devide ID check. Problem with ignoring device ID is, if file rotation is used (which you might still use in this case, so you can delete old logs + have filebeat process 'older' logs on rotation) you will have duplicates as well. We're considering some 'fingerprinting' on files or blocks of files to detect renames without having to rely on the device ID.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [November 6, 2017, 3:22am UTC](https://discuss.elastic.co/t/can-we-get-filebeats-to-optionally-ignore-deviceid-in-filestateos/105967/3 "2017-11-06T03:22:51Z")

</div>

Always interesting to see how new environment create new challenges 🙂 Is there something similar to the Device ID that stays the same?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 22, 2017, 12:43am UTC](https://discuss.elastic.co/t/can-we-get-filebeats-to-optionally-ignore-deviceid-in-filestateos/105967/4 "2017-11-22T00:43:19Z")

</div>

This topic was automatically closed after 21 days. New replies are no longer allowed.