# Use filebeat file output as logstash input

**URL:** <https://discuss.elastic.co/t/use-filebeat-file-output-as-logstash-input/187904>\
**Category:** Logstash\
**Created:** [June 27, 2019, 6:33pm UTC](https://discuss.elastic.co/t/use-filebeat-file-output-as-logstash-input/187904 "2019-06-27T18:33:28Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![maddin2016](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maddin2016/32/16599_2.png) [@maddin2016](https://discuss.elastic.co/u/maddin2016)\
**Post date:** [June 27, 2019, 6:33pm UTC](https://discuss.elastic.co/t/use-filebeat-file-output-as-logstash-input/187904/1 "2019-06-27T18:33:28Z")

</div>

I want to use the file output from filebeat as file input in logstash because there is no direct connection between the both systems. Here is my logstash config

```auto
input {
  file {
    path => "/home/martin/logs/filebeat*"
    type => "filebeat"
  }
}
output {
if [type] == "filebeat" {
  elasticsearch {
    hosts => "192.168.4.151:9200"
    index => "filebeat"
  }
}
}

```

Unfortunately it's not creating any index in elasticsearch. What it creates is only the `since_db` files. Also there no errors in the log files.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 27, 2019, 6:53pm UTC](https://discuss.elastic.co/t/use-filebeat-file-output-as-logstash-input/187904/2 "2019-06-27T18:53:04Z")

</div>

If you are using a remote mount make sure you understand the [limitations](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#_reading_from_remote_network_volumes).

Are you appending data to the files? The default value for [start\_position](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#plugins-inputs-file-start_position) is end.

---

<div class="post-metadata">

**Author:** ![maddin2016](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maddin2016/32/16599_2.png) [@maddin2016](https://discuss.elastic.co/u/maddin2016)\
**Post date:** [June 28, 2019, 6:25am UTC](https://discuss.elastic.co/t/use-filebeat-file-output-as-logstash-input/187904/3 "2019-06-28T06:25:33Z")

</div>

Yeap, that did the trick. I changed mode to `read`. Now i only have the problem that i always have to transfer the files every 30 minutes or so and only can do this with sftp. In filebeat i set rotation to every 20mb and a max of 40 files. Question now is what would be the best strategy on the logstash side. Let's assume i only transfer the latest `filebeat` file. What happens when i get the file with `19mb` and then in the next cycle because filebeat rotates to `filebeat.1` i get `filebeat` file with new data.

---

<div class="post-metadata">

**Author:** ![maddin2016](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maddin2016/32/16599_2.png) [@maddin2016](https://discuss.elastic.co/u/maddin2016)\
**Post date:** [July 1, 2019, 8:34am UTC](https://discuss.elastic.co/t/use-filebeat-file-output-as-logstash-input/187904/4 "2019-07-01T08:34:58Z")

</div>

I tried the solution to copy every time the filebeat logs to to the logstash server. But it seems that this will be end in a conflict with the inodes of the files and logstash every time read in the logs again. What would be the best solution in the case you have no direct connection between filebeat (or whatever log store) and logstash and you whant to import rotating files.

---

<div class="post-metadata">

**Author:** ![maddin2016](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maddin2016/32/16599_2.png) [@maddin2016](https://discuss.elastic.co/u/maddin2016)\
**Post date:** [July 2, 2019, 3:39pm UTC](https://discuss.elastic.co/t/use-filebeat-file-output-as-logstash-input/187904/5 "2019-07-02T15:39:14Z")

</div>

No one?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 2, 2019, 3:57pm UTC](https://discuss.elastic.co/t/use-filebeat-file-output-as-logstash-input/187904/6 "2019-07-02T15:57:35Z")

</div>

Tracking what subset of files you have read when those files can get rotated is an extremely hard problem. The file input does its best to track this using the inode and file size. If the way you are transferring files results in the inode not being a reliable way to track things, then the file input cannot do the tracking.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 30, 2019, 3:57pm UTC](https://discuss.elastic.co/t/use-filebeat-file-output-as-logstash-input/187904/7 "2019-07-30T15:57:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.