# How to close the file stream once the message has been pushed.?

**URL:** <https://discuss.elastic.co/t/how-to-close-the-file-stream-once-the-message-has-been-pushed/45983>\
**Category:** Logstash\
**Created:** [March 31, 2016, 6:26pm UTC](https://discuss.elastic.co/t/how-to-close-the-file-stream-once-the-message-has-been-pushed/45983 "2016-03-31T18:26:34Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![sdaruna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sdaruna/32/7289_2.png) [@sdaruna](https://discuss.elastic.co/u/sdaruna)\
**Post date:** [March 31, 2016, 6:26pm UTC](https://discuss.elastic.co/t/how-to-close-the-file-stream-once-the-message-has-been-pushed/45983/1 "2016-03-31T18:26:34Z")

</div>

Hi,

My requirement is to get the remote file using beats and write it to hdfs using logstash.

I need to write the output to hdfs using logstash. However, I cannot use webhdfs, so i used pipe output with hdfs appendToFile to push the data.

This is how i am passing the output.

```
input {
  beats {
    port => 5044
  }
}

filter {
  grok {   
    match => ["source","%{GREEDYDATA}/%{GREEDYDATA:filename}\.xml"]
  }
}

output {
  pipe { command => "hdfs dfs -appendToFile - /user/srini/%{filename}.xml" message_format => "%{message}"}
}

```

The hdfs files are getting opened and data has been written to it. However, the files are kept open with 0 bytes data until i stop logstash. I think it might be expecting more output. How can i role the file after the event.?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 31, 2016, 6:39pm UTC](https://discuss.elastic.co/t/how-to-close-the-file-stream-once-the-message-has-been-pushed/45983/2 "2016-03-31T18:39:12Z")

</div>

The pipe output's `ttl` option should with its default value close the pipe after 10 seconds of not receiving any events. Can you try enabling debug logging by starting Logstash with `--debug`? Then the plugin will log extra messages when it's closing the pipes.

---

<div class="post-metadata">

**Author:** ![sdaruna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sdaruna/32/7289_2.png) [@sdaruna](https://discuss.elastic.co/u/sdaruna)\
**Post date:** [March 31, 2016, 7:29pm UTC](https://discuss.elastic.co/t/how-to-close-the-file-stream-once-the-message-has-been-pushed/45983/3 "2016-03-31T19:29:08Z")

</div>

Hi @magnusbaeck,

I have added --debug and added ttl =\> 10 to just make sure it is 10 seconds.

There is no error apparently in the trace. Below is the stack trace.

```
Opening pipe {:command=>"hdfs dfs -appendToFile - /user/input2/srini.xml", :level=>:info, :file=>"logstash/outputs/pipe.rb", :line=>"100", :method=>"get_pipe"}
Pushing flush onto pipeline {:level=>:debug, :file=>"logstash/pipeline.rb", :line=>"450", :method=>"flush"}
16/03/31 15:25:09 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting stale pipes cleanup cycle {:pipes=>{"hdfs dfs -appendToFile - /user/netconf2/srini.xml"=>#<PipeWrapper:0x67ea2fc8 @pipe=#<IO:fd 483>, @active=true>}, :level=>:info, :file=>"logstash/outputs/pipe.rb", :line=>"75", :method=>"close_stale_pipes"}
0 stale pipes found {:inactive_pipes=>{}, :level=>:debug, :file=>"logstash/outputs/pipe.rb", :line=>"77", :method=>"close_stale_pipes"}
Pushing flush onto pipeline {:level=>:debug, :file=>"logstash/pipeline.rb", :line=>"450", :method=>"flush"}
Pushing flush onto pipeline {:level=>:debug, :file=>"logstash/pipeline.rb", :line=>"450", :method=>"flush"}
Pushing flush onto pipeline {:level=>:debug, :file=>"logstash/pipeline.rb", :line=>"450", :method=>"flush"}
Pushing flush onto pipeline {:level=>:debug, :file=>"logstash/pipeline.rb", :line=>"450", :method=>"flush"}
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 31, 2016, 8:20pm UTC](https://discuss.elastic.co/t/how-to-close-the-file-stream-once-the-message-has-been-pushed/45983/4 "2016-03-31T20:20:57Z")

</div>

Are you sure the pipe isn't getting events constantly? What if you replace the beats input with a stdin or something else where you control exactly which events are emitted?

(The log snippet above is not a stacktrace.)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:04am UTC](https://discuss.elastic.co/t/how-to-close-the-file-stream-once-the-message-has-been-pushed/45983/5 "2017-07-06T05:04:27Z")

</div>