# Regarding Filebeat Ouput to a File in Remote Server or NFS **URL:** \ **Category:** Beats\ **Created:** [April 1, 2016, 9:54am UTC](https://discuss.elastic.co/t/regarding-filebeat-ouput-to-a-file-in-remote-server-or-nfs/46038 "2016-04-01T09:54:58Z")\ **Posts on this page:** 19\ **Page:** 2
**Author:** ![vivc](https://avatars.discourse-cdn.com/v4/letter/v/ce7236/32.png) [@vivc](https://discuss.elastic.co/u/vivc)\ **Post date:** [May 2, 2016, 12:38pm UTC](https://discuss.elastic.co/t/regarding-filebeat-ouput-to-a-file-in-remote-server-or-nfs/46038/21 "2016-05-02T12:38:28Z")
Hi, I've tried the follwing filebeat --\> logstash --\> NFS -\> logstash --\> Elastic search. But after file beat is writing to the file in the format ```auto "message":"2016/03/23 06:34:10 [INFO] /RPSW1: Adding spanky(DEVICE)","@version":"1","@timestamp":"2016-04-25T15:11:20.290Z","beat":{"hostname":"Ericsson","name":"Ericsson"},"count":1,"fields":null,"input_type":"log","offset":0,"source":"/var/log/RPSW1_20160323063413.log","type":"log","host":"Ericsson","tags":["beats_input_codec_plain_applied"]} ``` when we store filebeat data to logstash into a file & again pass file data to logstash into elasticsearch, all the fields will be merged into one field which won’t serve our purpose. Reading the data from the file using the second logstash instance is not sending the lines correctly . I tried to use json\_lines as codec in logstash.conf and tried to read the filebeat data written to the file. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [May 2, 2016, 4:58pm UTC](https://discuss.elastic.co/t/regarding-filebeat-ouput-to-a-file-in-remote-server-or-nfs/46038/22 "2016-05-02T16:58:27Z")
it's quite hard to follow you. There are quite some parts in your setup. 1. What's the exact config per filebeat and logstash instances? 2. How does output look like in between every step? 3. instead of sending to elasticsearch for testing use stdout on last logstash instance. ---
**Author:** ![vivc](https://avatars.discourse-cdn.com/v4/letter/v/ce7236/32.png) [@vivc](https://discuss.elastic.co/u/vivc)\ **Post date:** [May 3, 2016, 4:07am UTC](https://discuss.elastic.co/t/regarding-filebeat-ouput-to-a-file-in-remote-server-or-nfs/46038/23 "2016-05-03T04:07:09Z")
Hi Steffens, There was some typo in the previous mail. I'm giving some sample format Format in which Filebeat write to the File: "message":"2016/03/23 06:34:10 [INFO] /RPSW1: Adding spanky(DEVICE)","@version":"1","@timestamp":"2016-04-25T15:11:20.290Z","beat":{"hostname":"Ericsson","name":"Ericsson"},"count":1,"fields":null,"input\_type":"log","offset":0,"source":"/var/log/RPSW1\_20160323063413.log","type":"log","host":"Ericsson","tags":["beats\_input\_codec\_plain\_applied"]} Format in which logstash read the data from the file: Logstash is taking the message as a single field. ![](https://us1.discourse-cdn.com/elastic/original/2X/b/b3810f687a8b1702b07b778b75ae76952c6fe5e7.png) It's taking a the line as a single filed and sending to elastic search . So our elastic search queries are not working. ---
**Author:** ![vivc](https://avatars.discourse-cdn.com/v4/letter/v/ce7236/32.png) [@vivc](https://discuss.elastic.co/u/vivc)\ **Post date:** [May 3, 2016, 4:08am UTC](https://discuss.elastic.co/t/regarding-filebeat-ouput-to-a-file-in-remote-server-or-nfs/46038/24 "2016-05-03T04:08:44Z")
Is there any difference in the way filebeat sending data to logstash directly and filebeat writing the file. and then logstash read the filebeat data from the file. ---
**Author:** ![vivc](https://avatars.discourse-cdn.com/v4/letter/v/ce7236/32.png) [@vivc](https://discuss.elastic.co/u/vivc)\ **Post date:** [May 3, 2016, 5:34am UTC](https://discuss.elastic.co/t/regarding-filebeat-ouput-to-a-file-in-remote-server-or-nfs/46038/25 "2016-05-03T05:34:17Z")
What's the exact config per filebeat and logstash instances? Filebeat output is configured as a file . Logstash is taking input from the file and forwarding it to elastic search . How does output look like in between every step? Logstash output to file: {"message":"root 19712 19709 0 02:08 ttyS0 00:00:00 grep dtpd","@timestamp":"2016-05-02T13:36:30.654Z","offset":366,"source":"/var/log/broncos\_logs/dvt\_1.txt","chs":"test","date":"02-05-2016"} Logstash output to elasticsearch: "message": "{"message":"root 19712 19709 0 02:08 ttyS0 00:00:00 grep dtpd","@timestamp":"2016-05-02T13:36:30.654Z","offset":366,"source":"/var/log/broncos\_logs/dvt\_1.txt","chs":"test","date":"02-05-2016"}" instead of sending to elasticsearch for testing use stdout on last logstash instance. We hvae checked the output in logstance instance it is showing as in the screen shot given in the previous reply. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 3, 2016, 7:14am UTC](https://discuss.elastic.co/t/regarding-filebeat-ouput-to-a-file-in-remote-server-or-nfs/46038/26 "2016-05-03T07:14:59Z")
The last Logstash instance which, as I understand it, reads a file with > {"message":"root 19712 19709 0 02:08 ttyS0 00:00:00 grep dtpd","@timestamp":"2016-05-02T13:36:30.654Z","offset":366,"source":"/var/log/broncos\_logs/dvt\_1.txt","chs":"test","date":"02-05-2016"} needs to use the json codec in its file input. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [May 3, 2016, 11:36am UTC](https://discuss.elastic.co/t/regarding-filebeat-ouput-to-a-file-in-remote-server-or-nfs/46038/27 "2016-05-03T11:36:44Z")
the log-output is pretty funny. it's a JSON event embedded into another JSON event. If filebeat is writing to file, the content is JSON. First logstash instance reading from file written by filebeat (why is this instance even required?) needs json codec in order to parse the event. ---
**Author:** ![vivc](https://avatars.discourse-cdn.com/v4/letter/v/ce7236/32.png) [@vivc](https://discuss.elastic.co/u/vivc)\ **Post date:** [May 4, 2016, 3:36am UTC](https://discuss.elastic.co/t/regarding-filebeat-ouput-to-a-file-in-remote-server-or-nfs/46038/28 "2016-05-04T03:36:03Z")
Yes , we tried using json code in input from file to second Ls instance but it's not working. As mentioned earlier we are trying to read from the file in NFS mount due to firewall issues. That is why we are using two instance. ---
**Author:** ![vivc](https://avatars.discourse-cdn.com/v4/letter/v/ce7236/32.png) [@vivc](https://discuss.elastic.co/u/vivc)\ **Post date:** [May 4, 2016, 3:36am UTC](https://discuss.elastic.co/t/regarding-filebeat-ouput-to-a-file-in-remote-server-or-nfs/46038/29 "2016-05-04T03:36:51Z")
But currently we have not tested reading from NFS mount file . We have tested reading from a local log file. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [May 4, 2016, 5:16am UTC](https://discuss.elastic.co/t/regarding-filebeat-ouput-to-a-file-in-remote-server-or-nfs/46038/30 "2016-05-04T05:16:56Z")
> Yes , we tried using json code in input from file to second Ls instance but it's not working. Exactly what did you try? What happened? Was there something in the logs? Can you provide a minimal example that exhibits the incorrect behavior? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [May 4, 2016, 2:41pm UTC](https://discuss.elastic.co/t/regarding-filebeat-ouput-to-a-file-in-remote-server-or-nfs/46038/31 "2016-05-04T14:41:24Z")
@vivc please provide input/output configs of filebeat and logstash instances. If warning/errors are printed by filebeat/logstash please include. To be honest, we've a hard time to even understand what exactly you are doing. I still don't understand why you need filebeat-\>logstash-\>NFS instead of filebeat-\>NFS. ---
**Author:** ![vivc](https://avatars.discourse-cdn.com/v4/letter/v/ce7236/32.png) [@vivc](https://discuss.elastic.co/u/vivc)\ **Post date:** [May 4, 2016, 2:56pm UTC](https://discuss.elastic.co/t/regarding-filebeat-ouput-to-a-file-in-remote-server-or-nfs/46038/32 "2016-05-04T14:56:11Z")
Sorry for that . Actually we are reading the data using filebeat and sending to logstash applying filters so that we get real data parsing time and sending it to file. Then the other logstash even which is installed in another server seperated by firewall will read the data from NFs file and store in elastic search. But the message written to file is shown as JSON event embedded into another JSON event. We have mentioned codec as json while reading the data from file to logstash. But still it's reading as a single field and sending to elastic search. Currently we are able to resolve it by applying the filters on the filebeat read data and getting the original message and again applying our filters on top of that . I'm not sure if you followed me , Thanks for your support. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [May 4, 2016, 4:11pm UTC](https://discuss.elastic.co/t/regarding-filebeat-ouput-to-a-file-in-remote-server-or-nfs/46038/33 "2016-05-04T16:11:24Z")
can you share config files? ---
**Author:** ![vivc](https://avatars.discourse-cdn.com/v4/letter/v/ce7236/32.png) [@vivc](https://discuss.elastic.co/u/vivc)\ **Post date:** [May 18, 2016, 10:28am UTC](https://discuss.elastic.co/t/regarding-filebeat-ouput-to-a-file-in-remote-server-or-nfs/46038/34 "2016-05-18T10:28:32Z")
Sorry for the delay for response. After pumping data from filebeat to file. The following grok pattern was used to retrieve the exact message. ``` grok { break_on_match => false match => { "message" => ["\"message\":\"(?.*)\"\,\"@timestamp\":\"(?.*)\"\,\"chassis\":\"(?.*)\"\,\"offset\":(?.*)\,\"source\":\"(?.*)\"" ] } } if [my_msg] { mutate { replace => ["message", "%{my_msg}"] remove_field => ["my_msg"] } } ``` Then the data is filtered correctly. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [May 18, 2016, 11:16am UTC](https://discuss.elastic.co/t/regarding-filebeat-ouput-to-a-file-in-remote-server-or-nfs/46038/35 "2016-05-18T11:16:24Z")
> [@vivc](#): > > After pumping data from filebeat to file. The following grok pattern was used to retrieve the exact message. Seriously, don't do this. Super fragile approach. filebeat stores data in json format. Use json filter to parse the filebeat output. Using grok on raw-json (without decoding first) might give you some headaches laters on, due to json potentially escaping strings. e.g. having a `"` in your original message might break your grok-pattern. Plus, json does not demand field order, but grok (basically being regexes on steroids) does. I'm a little baffled this works on filebeat restarts. I'd expect field order randomized between different runs. ---
**Author:** ![vivc](https://avatars.discourse-cdn.com/v4/letter/v/ce7236/32.png) [@vivc](https://discuss.elastic.co/u/vivc)\ **Post date:** [May 18, 2016, 11:24am UTC](https://discuss.elastic.co/t/regarding-filebeat-ouput-to-a-file-in-remote-server-or-nfs/46038/36 "2016-05-18T11:24:33Z")
Thanks for the information . We tried with json filter to parse the filebeat output.But that didn't work as i mentioned earlier. May we'll try it again and inform you . ---
**Author:** ![ausr12345](https://avatars.discourse-cdn.com/v4/letter/a/e68b1a/32.png) [@ausr12345](https://discuss.elastic.co/u/ausr12345)\ **Post date:** [June 21, 2016, 12:57pm UTC](https://discuss.elastic.co/t/regarding-filebeat-ouput-to-a-file-in-remote-server-or-nfs/46038/37 "2016-06-21T12:57:36Z")
Dear Magnus, could you please elaborate here? > That said, networked file systems are notorious for having different > edge case behavior and Logstash is usually not used for reading files > from NFS. We use a setup in which logstash consumes data from NFS share, therefore, would be very grateful for any further points on this. Thanks Joao ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [June 22, 2016, 6:07am UTC](https://discuss.elastic.co/t/regarding-filebeat-ouput-to-a-file-in-remote-server-or-nfs/46038/38 "2016-06-22T06:07:26Z")
Here are some notes for networks shares: [https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-network-volumes.html](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-network-volumes.html) The main problem for filebeat is that file identifiers can change even though it stays the same file. In addition sometimes file meta data is cached and not immidiately updated. It is recommended to install filebeat on each node. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 9:51pm UTC](https://discuss.elastic.co/t/regarding-filebeat-ouput-to-a-file-in-remote-server-or-nfs/46038/39 "2017-07-05T21:51:07Z")
Closed on July 5, 2017, 9:51pm UTC [Previous page](https://discuss.elastic.co/t/regarding-filebeat-ouput-to-a-file-in-remote-server-or-nfs/46038.md?page=1)