How do we fetch just the latest records from the input file and put them in output file?

Elastic Stack Beats
#1

Hi all,

Im trying to write the input data from input file to output file. I've put the below option in filebeats.yml file to copy the input to output.
output.file:
path: "/home/cloudera/Desktop"
filename: outB

But the problem is, for the first time I have one record "rec-1". When I run the filebeats, it is moving the records from input to output. My output file now contains: "rec-1"
But when enter another record in the input "rec-2", it is taking "rec-1, rec-2" from the input file and updating the output file. My output file now contains: "rec-1", "rec-1", "rec-2"
Can anyone tell me if there is any way to take just the latest updated records and put only those records in the output file ?
in mycase: my output file should have only: "rec-1","rec-2".
Thanks in advance.

#2
  • Can you share your full filebeat config? Please use 3 ` (ticks) around the config to preserver indenation
  • Please share your filebeat log output
  • How do you update the file to append content?
  • Which filebeat version are you using?
#4

[quote="bobbysidhartha, post:3, topic:80240, full:true"]

  1. YML file looks like:#=========================== Filebeat prospectors =============================filebeat.prospectors:

  • input_type: log

    Paths that should be crawled and fetched. Glob based paths.

    paths:

    • /home/cloudera/Desktop/inpB
      #- /var/log/*.log
      #- c:\programdata\elasticsearch\logs*
      #=========================== Filebeat prospectors =============================
      #----------------------------- Writing output file-----------------------------
      output.file:
      path: "/home/cloudera/Desktop"
      filename: outBechnique.
      #----------------------------- Writing output file-----------------------------
  1. Currently my input file contain 4 lines: rec, rec2, rec3, rec4
    Output file:{"@timestamp":"2017-03-28T07:17:58.910Z","beat":{"hostname":"quickstart.cloudera","name":"quickstart.cloudera","version":"5.2.2"},"input_type":"log","message":"rec","offset":4,"source":"/home/cloudera/Desktop/inpB","type":"log"}{"@timestamp":"2017-03-28T07:17:58.910Z","beat":{"hostname":"quickstart.cloudera","name":"quickstart.cloudera","version":"5.2.2"},"input_type":"log","message":"rec2","offset":9,"source":"/home/cloudera/Desktop/inpB","type":"log"}{"@timestamp":"2017-03-28T07:17:58.910Z","beat":{"hostname":"quickstart.cloudera","name":"quickstart.cloudera","version":"5.2.2"},"input_type":"log","message":"rec3","offset":14,"source":"/home/cloudera/Desktop/inpB","type":"log"}{"@timestamp":"2017-03-28T07:25:38.982Z","beat":{"hostname":"quickstart.cloudera","name":"quickstart.cloudera","version":"5.2.2"},"input_type":"log","message":"rec","offset":4,"source":"/home/cloudera/Desktop/inpB","type":"log"}{"@timestamp":"2017-03-28T07:25:38.982Z","beat":{"hostname":"quickstart.cloudera","name":"quickstart.cloudera","version":"5.2.2"},"input_type":"log","message":"rec2","offset":9,"source":"/home/cloudera/Desktop/inpB","type":"log"}{"@timestamp":"2017-03-28T07:25:38.982Z","beat":{"hostname":"quickstart.cloudera","name":"quickstart.cloudera","version":"5.2.2"},"input_type":"log","message":"rec3","offset":14,"source":"/home/cloudera/Desktop/inpB","type":"log"}{"@timestamp":"2017-03-28T07:25:38.982Z","beat":{"hostname":"quickstart.cloudera","name":"quickstart.cloudera","version":"5.2.2"},"input_type":"log","message":"rec4","offset":19,"source":"/home/cloudera/Desktop/inpB","type":"log"}{"@timestamp":"2017-03-28T07:25:58.985Z","beat":{"hostname":"quickstart.cloudera","name":"quickstart.cloudera","version":"5.2.2"},"input_type":"log","message":"rec","offset":4,"source":"/home/cloudera/Desktop/inpB","type":"log"}{"@timestamp":"2017-03-28T07:25:58.985Z","beat":{"hostname":"quickstart.cloudera","name":"quickstart.cloudera","version":"5.2.2"},"input_type":"log","message":"rec2","offset":9,"source":"/home/cloudera/Desktop/inpB","type":"log"}{"@timestamp":"2017-03-28T07:25:58.985Z","beat":{"hostname":"quickstart.cloudera","name":"quickstart.cloudera","version":"5.2.2"},"input_type":"log","message":"rec3","offset":14,"source":"/home/cloudera/Desktop/inpB","type":"log"}`{"@timestamp":"2017-03-28T07:25:58.985Z","beat":{"hostname":"quickstart.cloudera","name":"quickstart.cloudera","version":"5.2.2"},"input_type":"log","message":"rec4","offset":19,"source":"/home/cloudera/Desktop/inpB","type":"log"}
  2. Im asking for any suggestion. Im trying to append the data but Im failing as I don't know the proper the proper technique.
  3. The version Im using is: 5.2.2
#5

Can you try for format your post above? Or otherwise can you create a gist and link it here?

#6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.