# Filebeat from type "log" to "filestream" duplicated events

**URL:** https://discuss.elastic.co/t/filebeat-from-type-log-to-filestream-duplicated-events/364268
**Category:** Beats
**Tags:** filebeat
**Created:** [August 2, 2024, 10:19am UTC](https://discuss.elastic.co/t/filebeat-from-type-log-to-filestream-duplicated-events/364268 "2024-08-02T10:19:49Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![t.h](https://avatars.discourse-cdn.com/v4/letter/t/b5e925/32.png) [@t.h](https://discuss.elastic.co/u/t.h)
#### Post date: [August 2, 2024, 10:19am UTC](https://discuss.elastic.co/t/filebeat-from-type-log-to-filestream-duplicated-events/364268/1 "2024-08-02T10:19:49Z")

</div>

Hello,

we are changing from the filebeat type "log" to "filesteam"  
Since we are changed, we have duplicates events in our logstash. Looks like this was the reason of my other problem with the logstash throttle and max\_age time. Because the old log events are pushed some days later again.

We are not changing the default `file_identity` settings

Old config:

```auto
- type: log
  paths:
    - /var/log/commserverSpool.log
  fields:
   type: commserver_spool
   application: '${APPLICATION}'
   apptype: dc
  fields_under_root: true
  symlinks: true

```

New config:

```auto
- type: filestream
  id: '${FBHOST}:commserver_spool'
  paths:
    - /var/log/commserverSpool.log
  fields:
   type: commserver_spool
   application: '${APPLICATION}'
   apptype: dc
  fields_under_root: true
  symlinks: true

```

Any idea what I make wrong?

Thx  
Thomas

---

<div class="post-metadata">

### Author: ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [August 2, 2024, 1:29pm UTC](https://discuss.elastic.co/t/filebeat-from-type-log-to-filestream-duplicated-events/364268/2 "2024-08-02T13:29:55Z")

</div>

Which version are you using?

The `filestream` input will work as a new input, so it will per default read the files from the start as it store the registry in a different way, so one of the issues of just changing `log` to `filestream` is that it can lead to duplication.

There is however an new option named `take_over` to avoid this, not sure in which version this was introduced, but to migrate from a `log` input to a `filestream` input you need to follow the steps described here: [migrate from log to filestream](https://www.elastic.co/guide/en/beats/filebeat/current/migrate-to-filestream.html).

---

<div class="post-metadata">

### Author: ![t.h](https://avatars.discourse-cdn.com/v4/letter/t/b5e925/32.png) [@t.h](https://discuss.elastic.co/u/t.h)
#### Post date: [August 2, 2024, 1:57pm UTC](https://discuss.elastic.co/t/filebeat-from-type-log-to-filestream-duplicated-events/364268/3 "2024-08-02T13:57:39Z")

</div>

Thank you for your response

We use this filebeat version:

```auto
filebeat version
filebeat version 8.14.3 (amd64), libbeat 8.14.3 [71819961045386b23edc18455f1b54764292816c built 2024-07-08 22:05:44 +0000 UTC]

```

The server is a new installed RHEL9 server. All log files are new. Is not really a migration, only the config. We are starting direct with the new `filestream` config. But the logs are repeating. The file is not rotating.

Now I make a rollback to the old type `log`, with this config we don't have any problems. Sure, on the first read we have duplicates, but then not anymore.

I know `log` is deprecated, this is the reason why I want to understand our problem.

---

<div class="post-metadata">

### Author: ![t.h](https://avatars.discourse-cdn.com/v4/letter/t/b5e925/32.png) [@t.h](https://discuss.elastic.co/u/t.h)
#### Post date: [August 7, 2024, 11:24am UTC](https://discuss.elastic.co/t/filebeat-from-type-log-to-filestream-duplicated-events/364268/4 "2024-08-07T11:24:59Z")

</div>

Now I go back to `log` without any problems. Any ideas how I can solve the problem?