Data getting duplicated - rsync'ing output to Logstash file input


(Rajat Singh) #1

i am pulling logs from remote server using rsync command in every 5 minutes and then feeding it to the logstash

what is happening that it is just creating the duplicate logs of it


(Andrew Kroh) #2

Can you provide some more details on your setup. What is the Topbeat and Logstash configuration your are using? What is the full rsync command your using to sync logs?

Why not use the Logstash output that is built into Topbeat?


(Steffen Siering) #3

This sounds like rsync is creating a very new file instead of appending when transfering/syncing logs.


(Rajat Singh) #4

rsync does not create the new file it will just append the data
http://everythinglinux.org/rsync/

you can refer above link for the same


(Rajat Singh) #5

My logtash file is running continously but it is duplicating the logs 3 or 4 times only and everytime it is having unique document id also


(Andrew Kroh) #6

You can check the inode of the file with stat before and after the rsync to see if it is actually the same file.

What happens when the Topbeat output file gets rotated? Could that be causing problems.


(Rajat Singh) #7

inode is changing i had checked it


(Magnus B├Ąck) #8

You need to get rsync to append the data or find another method of transferring the files.


(Rajat Singh) #9

is there any other method that only appends the data ?


(Steffen Siering) #10

inode changing is a sign of rsync actually not appending, but actually creating a new file. I'm not really firm with rsync myself, but for people having some more knowledge, it would be helpful if you can share details on how rsync is configured. That some more details on config files and or command line flags.


(Rajat Singh) #11

thanks for your support i got it
--inplace --no-whole-file

you had to add it with rsync inoder not create the new file everytime


(system) #12

This topic was automatically closed after 21 days. New replies are no longer allowed.