# How do we fetch just the latest records from the input file and put them in output file? **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [March 28, 2017, 5:35am UTC](https://discuss.elastic.co/t/how-do-we-fetch-just-the-latest-records-from-the-input-file-and-put-them-in-output-file/80240 "2017-03-28T05:35:39Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![bobbysidhartha](https://avatars.discourse-cdn.com/v4/letter/b/3ab097/32.png) [@bobbysidhartha](https://discuss.elastic.co/u/bobbysidhartha)\ **Post date:** [March 28, 2017, 5:35am UTC](https://discuss.elastic.co/t/how-do-we-fetch-just-the-latest-records-from-the-input-file-and-put-them-in-output-file/80240/1 "2017-03-28T05:35:39Z")
Hi all, Im trying to write the input data from input file to output file. I've put the below option in filebeats.yml file to copy the input to output. output.file: path: "/home/cloudera/Desktop" filename: outB But the problem is, for the first time I have one record "rec-1". When I run the filebeats, it is moving the records from input to output. My output file now contains: "rec-1" But when enter another record in the input "rec-2", it is taking "rec-1, rec-2" from the input file and updating the output file. My output file now contains: "rec-1", "rec-1", "rec-2" Can anyone tell me if there is any way to take just the latest updated records and put only those records in the output file ? in mycase: my output file should have only: "rec-1","rec-2". Thanks in advance. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [March 29, 2017, 6:52am UTC](https://discuss.elastic.co/t/how-do-we-fetch-just-the-latest-records-from-the-input-file-and-put-them-in-output-file/80240/2 "2017-03-29T06:52:30Z")
- Can you share your full filebeat config? Please use 3 ` (ticks) around the config to preserver indenation - Please share your filebeat log output - How do you update the file to append content? - Which filebeat version are you using? ---
**Author:** ![bobbysidhartha](https://avatars.discourse-cdn.com/v4/letter/b/3ab097/32.png) [@bobbysidhartha](https://discuss.elastic.co/u/bobbysidhartha)\ **Post date:** [March 30, 2017, 7:06am UTC](https://discuss.elastic.co/t/how-do-we-fetch-just-the-latest-records-from-the-input-file-and-put-them-in-output-file/80240/4 "2017-03-30T07:06:28Z")
[quote="bobbysidhartha, post:3, topic:80240, full:true"] 1. YML file looks like:#=========================== Filebeat prospectors =============================filebeat.prospectors: - input\_type: log 1. Currently my input file contain 4 lines: rec, rec2, rec3, rec4 Output file:{"@timestamp":"2017-03-28T07:17:58.910Z","beat":{"hostname":"quickstart.cloudera","name":"quickstart.cloudera","version":"5.2.2"},"input\_type":"log","message":"rec","offset":4,"source":"/home/cloudera/Desktop/inpB","type":"log"}{"@timestamp":"2017-03-28T07:17:58.910Z","beat":{"hostname":"quickstart.cloudera","name":"quickstart.cloudera","version":"5.2.2"},"input\_type":"log","message":"rec2","offset":9,"source":"/home/cloudera/Desktop/inpB","type":"log"}{"@timestamp":"2017-03-28T07:17:58.910Z","beat":{"hostname":"quickstart.cloudera","name":"quickstart.cloudera","version":"5.2.2"},"input\_type":"log","message":"rec3","offset":14,"source":"/home/cloudera/Desktop/inpB","type":"log"}{"@timestamp":"2017-03-28T07:25:38.982Z","beat":{"hostname":"quickstart.cloudera","name":"quickstart.cloudera","version":"5.2.2"},"input\_type":"log","message":"rec","offset":4,"source":"/home/cloudera/Desktop/inpB","type":"log"}{"@timestamp":"2017-03-28T07:25:38.982Z","beat":{"hostname":"quickstart.cloudera","name":"quickstart.cloudera","version":"5.2.2"},"input\_type":"log","message":"rec2","offset":9,"source":"/home/cloudera/Desktop/inpB","type":"log"}{"@timestamp":"2017-03-28T07:25:38.982Z","beat":{"hostname":"quickstart.cloudera","name":"quickstart.cloudera","version":"5.2.2"},"input\_type":"log","message":"rec3","offset":14,"source":"/home/cloudera/Desktop/inpB","type":"log"}{"@timestamp":"2017-03-28T07:25:38.982Z","beat":{"hostname":"quickstart.cloudera","name":"quickstart.cloudera","version":"5.2.2"},"input\_type":"log","message":"rec4","offset":19,"source":"/home/cloudera/Desktop/inpB","type":"log"}{"@timestamp":"2017-03-28T07:25:58.985Z","beat":{"hostname":"quickstart.cloudera","name":"quickstart.cloudera","version":"5.2.2"},"input\_type":"log","message":"rec","offset":4,"source":"/home/cloudera/Desktop/inpB","type":"log"}{"@timestamp":"2017-03-28T07:25:58.985Z","beat":{"hostname":"quickstart.cloudera","name":"quickstart.cloudera","version":"5.2.2"},"input\_type":"log","message":"rec2","offset":9,"source":"/home/cloudera/Desktop/inpB","type":"log"}{"@timestamp":"2017-03-28T07:25:58.985Z","beat":{"hostname":"quickstart.cloudera","name":"quickstart.cloudera","version":"5.2.2"},"input\_type":"log","message":"rec3","offset":14,"source":"/home/cloudera/Desktop/inpB","type":"log"}`{"@timestamp":"2017-03-28T07:25:58.985Z","beat":{"hostname":"quickstart.cloudera","name":"quickstart.cloudera","version":"5.2.2"},"input\_type":"log","message":"rec4","offset":19,"source":"/home/cloudera/Desktop/inpB","type":"log"} 2. Im asking for any suggestion. Im trying to append the data but Im failing as I don't know the proper the proper technique. 3. The version Im using is: 5.2.2 ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [March 30, 2017, 11:36am UTC](https://discuss.elastic.co/t/how-do-we-fetch-just-the-latest-records-from-the-input-file-and-put-them-in-output-file/80240/5 "2017-03-30T11:36:58Z")
Can you try for format your post above? Or otherwise can you create a gist and link it here? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 27, 2017, 11:37am UTC](https://discuss.elastic.co/t/how-do-we-fetch-just-the-latest-records-from-the-input-file-and-put-them-in-output-file/80240/6 "2017-04-27T11:37:21Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.