# Data getting duplicated - rsync'ing output to Logstash file input **URL:** \ **Category:** Beats\ **Created:** [November 3, 2016, 3:05pm UTC](https://discuss.elastic.co/t/data-getting-duplicated-rsyncing-output-to-logstash-file-input/64886 "2016-11-03T15:05:01Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Rajat\_Singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rajat_singh/32/48128_2.png) [@Rajat\_Singh](https://discuss.elastic.co/u/Rajat_Singh)\ **Post date:** [November 3, 2016, 3:05pm UTC](https://discuss.elastic.co/t/data-getting-duplicated-rsyncing-output-to-logstash-file-input/64886/1 "2016-11-03T15:05:01Z")
i am pulling logs from remote server using rsync command in every 5 minutes and then feeding it to the logstash what is happening that it is just creating the duplicate logs of it ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [November 3, 2016, 4:02pm UTC](https://discuss.elastic.co/t/data-getting-duplicated-rsyncing-output-to-logstash-file-input/64886/2 "2016-11-03T16:02:52Z")
Can you provide some more details on your setup. What is the Topbeat and Logstash configuration your are using? What is the full rsync command your using to sync logs? Why not use the Logstash output that is built into Topbeat? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 3, 2016, 9:55pm UTC](https://discuss.elastic.co/t/data-getting-duplicated-rsyncing-output-to-logstash-file-input/64886/3 "2016-11-03T21:55:34Z")
This sounds like rsync is creating a very new file instead of appending when transfering/syncing logs. ---
**Author:** ![Rajat\_Singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rajat_singh/32/48128_2.png) [@Rajat\_Singh](https://discuss.elastic.co/u/Rajat_Singh)\ **Post date:** [November 4, 2016, 2:28pm UTC](https://discuss.elastic.co/t/data-getting-duplicated-rsyncing-output-to-logstash-file-input/64886/4 "2016-11-04T14:28:50Z")
rsync does not create the new file it will just append the data [http://everythinglinux.org/rsync/](http://everythinglinux.org/rsync/) you can refer above link for the same ---
**Author:** ![Rajat\_Singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rajat_singh/32/48128_2.png) [@Rajat\_Singh](https://discuss.elastic.co/u/Rajat_Singh)\ **Post date:** [November 4, 2016, 2:33pm UTC](https://discuss.elastic.co/t/data-getting-duplicated-rsyncing-output-to-logstash-file-input/64886/5 "2016-11-04T14:33:16Z")
My logtash file is running continously but it is duplicating the logs 3 or 4 times only and everytime it is having unique document id also ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [November 4, 2016, 2:38pm UTC](https://discuss.elastic.co/t/data-getting-duplicated-rsyncing-output-to-logstash-file-input/64886/6 "2016-11-04T14:38:54Z")
You can check the inode of the file with `stat` before and after the `rsync` to see if it is actually the same file. What happens when the Topbeat output file gets rotated? Could that be causing problems. ---
**Author:** ![Rajat\_Singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rajat_singh/32/48128_2.png) [@Rajat\_Singh](https://discuss.elastic.co/u/Rajat_Singh)\ **Post date:** [November 4, 2016, 2:56pm UTC](https://discuss.elastic.co/t/data-getting-duplicated-rsyncing-output-to-logstash-file-input/64886/7 "2016-11-04T14:56:10Z")
inode is changing i had checked it ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [November 4, 2016, 3:29pm UTC](https://discuss.elastic.co/t/data-getting-duplicated-rsyncing-output-to-logstash-file-input/64886/8 "2016-11-04T15:29:18Z")
You need to get rsync to append the data or find another method of transferring the files. ---
**Author:** ![Rajat\_Singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rajat_singh/32/48128_2.png) [@Rajat\_Singh](https://discuss.elastic.co/u/Rajat_Singh)\ **Post date:** [November 4, 2016, 3:51pm UTC](https://discuss.elastic.co/t/data-getting-duplicated-rsyncing-output-to-logstash-file-input/64886/9 "2016-11-04T15:51:00Z")
is there any other method that only appends the data ? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [November 4, 2016, 11:49pm UTC](https://discuss.elastic.co/t/data-getting-duplicated-rsyncing-output-to-logstash-file-input/64886/10 "2016-11-04T23:49:16Z")
inode changing is a sign of rsync actually not appending, but actually creating a new file. I'm not really firm with rsync myself, but for people having some more knowledge, it would be helpful if you can share details on how rsync is configured. That some more details on config files and or command line flags. ---
**Author:** ![Rajat\_Singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rajat_singh/32/48128_2.png) [@Rajat\_Singh](https://discuss.elastic.co/u/Rajat_Singh)\ **Post date:** [November 7, 2016, 6:07am UTC](https://discuss.elastic.co/t/data-getting-duplicated-rsyncing-output-to-logstash-file-input/64886/11 "2016-11-07T06:07:18Z")
thanks for your support i got it --inplace --no-whole-file you had to add it with rsync inoder not create the new file everytime ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 24, 2016, 3:05pm UTC](https://discuss.elastic.co/t/data-getting-duplicated-rsyncing-output-to-logstash-file-input/64886/12 "2016-11-24T15:05:20Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.