Data getting duplicated - rsync'ing output to Logstash file input

Elastic Stack Beats
1

i am pulling logs from remote server using rsync command in every 5 minutes and then feeding it to the logstash

what is happening that it is just creating the duplicate logs of it

2

Can you provide some more details on your setup. What is the Topbeat and Logstash configuration your are using? What is the full rsync command your using to sync logs?

Why not use the Logstash output that is built into Topbeat?

1 Like
3

This sounds like rsync is creating a very new file instead of appending when transfering/syncing logs.

4

rsync does not create the new file it will just append the data
http://everythinglinux.org/rsync/

you can refer above link for the same

5

My logtash file is running continously but it is duplicating the logs 3 or 4 times only and everytime it is having unique document id also

6

You can check the inode of the file with stat before and after the rsync to see if it is actually the same file.

What happens when the Topbeat output file gets rotated? Could that be causing problems.

1 Like
7

inode is changing i had checked it

8

You need to get rsync to append the data or find another method of transferring the files.

1 Like
9

is there any other method that only appends the data ?

10

inode changing is a sign of rsync actually not appending, but actually creating a new file. I'm not really firm with rsync myself, but for people having some more knowledge, it would be helpful if you can share details on how rsync is configured. That some more details on config files and or command line flags.

11

thanks for your support i got it
--inplace --no-whole-file

you had to add it with rsync inoder not create the new file everytime

4 Likes
12

This topic was automatically closed after 21 days. New replies are no longer allowed.