i am pulling logs from remote server using rsync command in every 5 minutes and then feeding it to the logstash
what is happening that it is just creating the duplicate logs of it
Can you provide some more details on your setup. What is the Topbeat and Logstash configuration your are using? What is the full rsync command your using to sync logs?
Why not use the Logstash output that is built into Topbeat?
1 Like
This sounds like rsync is creating a very new file instead of appending when transfering/syncing logs.
rsync does not create the new file it will just append the data
http://everythinglinux.org/rsync/
you can refer above link for the same
My logtash file is running continously but it is duplicating the logs 3 or 4 times only and everytime it is having unique document id also
You can check the inode of the file with stat before and after the rsync to see if it is actually the same file.
What happens when the Topbeat output file gets rotated? Could that be causing problems.
1 Like
inode is changing i had checked it
You need to get rsync to append the data or find another method of transferring the files.
1 Like
is there any other method that only appends the data ?
inode changing is a sign of rsync actually not appending, but actually creating a new file. I'm not really firm with rsync myself, but for people having some more knowledge, it would be helpful if you can share details on how rsync is configured. That some more details on config files and or command line flags.
thanks for your support i got it
--inplace --no-whole-file
you had to add it with rsync inoder not create the new file everytime
4 Likes
This topic was automatically closed after 21 days. New replies are no longer allowed.