# Multiple logstash nodes use file-output plugin to output messages to one file in a shared file system? **URL:** \ **Category:** Logstash\ **Created:** [July 24, 2017, 6:32am UTC](https://discuss.elastic.co/t/multiple-logstash-nodes-use-file-output-plugin-to-output-messages-to-one-file-in-a-shared-file-system/94275 "2017-07-24T06:32:42Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![KeithTt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keithtt/32/29447_2.png) [@KeithTt](https://discuss.elastic.co/u/KeithTt)\ **Post date:** [July 24, 2017, 6:32am UTC](https://discuss.elastic.co/t/multiple-logstash-nodes-use-file-output-plugin-to-output-messages-to-one-file-in-a-shared-file-system/94275/1 "2017-07-24T06:32:42Z")
ELK version: 5.4.1 There are 3 logstash nodes in my BELK arch, and I configure load-balancing in filebeat. if I use output-plugin to output the messages from 3 logstash nodes to a same file in a shared filesystem, will the new messages overwrite the old ones or will that be conflict? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 24, 2017, 6:40am UTC](https://discuss.elastic.co/t/multiple-logstash-nodes-use-file-output-plugin-to-output-messages-to-one-file-in-a-shared-file-system/94275/2 "2017-07-24T06:40:03Z")
As there is no coordination around file writing between the Logstash instances, I would expect writing to a shared file to cause serious problems. ---
**Author:** ![KeithTt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keithtt/32/29447_2.png) [@KeithTt](https://discuss.elastic.co/u/KeithTt)\ **Post date:** [July 24, 2017, 6:44am UTC](https://discuss.elastic.co/t/multiple-logstash-nodes-use-file-output-plugin-to-output-messages-to-one-file-in-a-shared-file-system/94275/3 "2017-07-24T06:44:39Z")
Yes, this what I was thinking about... So....I need to confirm with you that will this situation cause problem? And, do you have a available method to do this? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 24, 2017, 6:49am UTC](https://discuss.elastic.co/t/multiple-logstash-nodes-use-file-output-plugin-to-output-messages-to-one-file-in-a-shared-file-system/94275/4 "2017-07-24T06:49:18Z")
I can not see how it would work, so am reasonably sure it will cause problems. I have however never tested it. The only way to do this is, as far as I know, to send data to a single Logstash instance that does all the writing. ---
**Author:** ![KeithTt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keithtt/32/29447_2.png) [@KeithTt](https://discuss.elastic.co/u/KeithTt)\ **Post date:** [July 24, 2017, 7:06am UTC](https://discuss.elastic.co/t/multiple-logstash-nodes-use-file-output-plugin-to-output-messages-to-one-file-in-a-shared-file-system/94275/5 "2017-07-24T07:06:27Z")
OK, I see... By the way, it seems that there can define only one file-output in one pipeline config file? and this file will record all the events even if I use `if` statement. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 24, 2017, 8:07am UTC](https://discuss.elastic.co/t/multiple-logstash-nodes-use-file-output-plugin-to-output-messages-to-one-file-in-a-shared-file-system/94275/6 "2017-07-24T08:07:29Z")
I am not sure I understand. Can you show an example of what you are trying to do? ---
**Author:** ![KeithTt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keithtt/32/29447_2.png) [@KeithTt](https://discuss.elastic.co/u/KeithTt)\ **Post date:** [July 24, 2017, 9:53am UTC](https://discuss.elastic.co/t/multiple-logstash-nodes-use-file-output-plugin-to-output-messages-to-one-file-in-a-shared-file-system/94275/7 "2017-07-24T09:53:00Z")
here is part of my pipeline config file: ```auto input { beats { port => 5044 codec => "json" } } output { if [type] == "zixun-nginx-access" { elasticsearch { hosts => ["192.168.3.56:9200","192.168.3.49:9200","192.168.3.57:9200"] index => "zixun-nginx-access-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" template_overwrite => true }} file { path => "/nh/esbk/my_backup/backup/kibana-nginx-access-%{+YYYY-MM-dd}.log" } if [type] == "water-nginx-access" { elasticsearch { hosts => ["192.168.3.56:9200","192.168.3.49:9200","192.168.3.57:9200"] index => "water-nginx-access-%{+YYYY.MM.dd}" document_type => "%{[@metadata][type]}" template_overwrite => true }} ... } ``` I desire to output messages to different file for every `type`. ---
**Author:** ![KeithTt](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/keithtt/32/29447_2.png) [@KeithTt](https://discuss.elastic.co/u/KeithTt)\ **Post date:** [July 25, 2017, 3:25am UTC](https://discuss.elastic.co/t/multiple-logstash-nodes-use-file-output-plugin-to-output-messages-to-one-file-in-a-shared-file-system/94275/8 "2017-07-25T03:25:05Z")
It seems that I should define multiple pipeline config files for different projects and startup multiple logstash instances on each logstash node? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 26, 2017, 7:41am UTC](https://discuss.elastic.co/t/multiple-logstash-nodes-use-file-output-plugin-to-output-messages-to-one-file-in-a-shared-file-system/94275/9 "2017-07-26T07:41:31Z")
> As there is no coordination around file writing between the Logstash instances, I would expect writing to a shared file to cause serious problems. If Logstash is opening the output file with O\_APPEND (which it should) then all write() operations will be made at the end of the file even if multiple processes write to the file concurrently. See [write](http://pubs.opengroup.org/onlinepubs/9699919799/functions/write.html). This doesn't apply to Windows though. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 23, 2017, 7:41am UTC](https://discuss.elastic.co/t/multiple-logstash-nodes-use-file-output-plugin-to-output-messages-to-one-file-in-a-shared-file-system/94275/10 "2017-08-23T07:41:45Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.