# Replication in ES **URL:** \ **Category:** Logstash\ **Created:** [June 19, 2017, 5:43am UTC](https://discuss.elastic.co/t/replication-in-es/89882 "2017-06-19T05:43:22Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![raghvendra](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@raghvendra](https://discuss.elastic.co/u/raghvendra)\ **Post date:** [June 19, 2017, 5:43am UTC](https://discuss.elastic.co/t/replication-in-es/89882/1 "2017-06-19T05:43:22Z")
Hey .. can anyone make one thing clear .. i was running ELK stack. Logstash was reading files from a logs from a folder on my local machine. I had to stop the logstash for some reason and then i had restarted it .. everything is going fine . But since I have restarted it , will it start reading files from the starting . If it will , it means there would me replications of same documents with different document id because they might have got read by logstash twice or it will not let replication happen . In fact ES does replicate each shard , I am not talking about those replications . Any help would be appreciated . ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 19, 2017, 2:29pm UTC](https://discuss.elastic.co/t/replication-in-es/89882/2 "2017-06-19T14:29:09Z")
> Logstash was reading files from a logs from a folder on my local machine. I had to stop the logstash for some reason and then i had restarted it .. everything is going fine . But since I have restarted it , will it start reading files from the starting . It won't if it's correctly configured. There are ways to screw this up but the default configuration is safe in this regard. ---
**Author:** ![Ranjith\_M](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ranjith_m/32/19272_2.png) [@Ranjith\_M](https://discuss.elastic.co/u/Ranjith_M)\ **Post date:** [June 19, 2017, 3:20pm UTC](https://discuss.elastic.co/t/replication-in-es/89882/3 "2017-06-19T15:20:10Z")
> [@magnusbaeck](#): > > It won't if it's correctly configured. There are ways to screw this up but the default configuration is safe in this regard. Hi Magnus, Where does logstash store it's meta information, about what it processed. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 19, 2017, 8:57pm UTC](https://discuss.elastic.co/t/replication-in-es/89882/4 "2017-06-19T20:57:08Z")
[https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#\_tracking\_of\_current\_position\_in\_watched\_files](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html#_tracking_of_current_position_in_watched_files) ---
**Author:** ![raghvendra](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@raghvendra](https://discuss.elastic.co/u/raghvendra)\ **Post date:** [June 30, 2017, 11:44am UTC](https://discuss.elastic.co/t/replication-in-es/89882/5 "2017-06-30T11:44:59Z")
@magnusbaeck I am putting my old data into ES , so I run logstash durning the office hours and then I stop and do the same in next morning . By opening this topic and by your answer that if you stick to the default configuration, it won't make any replicas of a document with different document id. I was doing the same. But when i crossed check , and applied aggregation on userid and timestamp , idealy, doc\_count should be zero because timestamp can't be same for the same user .But it is showing some value , it means replications did happen . And through kibana , I verified it. Here is the screenshot - ![](https://us1.discourse-cdn.com/elastic/original/3X/4/4/44b92a4d14399e565edab1423450c0a358ac2b01.png) see first two documents are identical with different document id . Can you tell me how and the solution to prevent it happening again ? I am using default configuration of logstash . ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [June 30, 2017, 1:08pm UTC](https://discuss.elastic.co/t/replication-in-es/89882/6 "2017-06-30T13:08:39Z")
It's impossible for me to tell why this happened. The evidence (the sincedb file) has been overwritten and complete logs aren't available. ---
**Author:** ![raghvendra](https://avatars.discourse-cdn.com/v4/letter/r/c68b51/32.png) [@raghvendra](https://discuss.elastic.co/u/raghvendra)\ **Post date:** [July 5, 2017, 12:24pm UTC](https://discuss.elastic.co/t/replication-in-es/89882/7 "2017-07-05T12:24:29Z")
@magnusbaeck Hi... if I put this in logstash's config file .. > input{ > file { > path =\> "/home/mywavia/new/accesslog49.txt" > start\_position =\> "beginning" > } and i start it after stopping it for some reason , would logstash start parsing the input file from the beginning each time I restart ??, Because , I had a input file with 1.6 million documents , after parsing through logstash , I found some 4 million documents in that particular index through kibana's monitoring section . Do you think that "start\_position" tag made logstash to start over on the same input file each time after I restarted it ? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [July 5, 2017, 2:31pm UTC](https://discuss.elastic.co/t/replication-in-es/89882/8 "2017-07-05T14:31:32Z")
> and i start it after stopping it for some reason , would logstash start parsing the input file from the beginning each time I restart ??, No, it'll still use your sincedb file. The `start_position` option only matters for previously unseen files, i.e. files with no sincedb entry. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 2, 2017, 2:31pm UTC](https://discuss.elastic.co/t/replication-in-es/89882/9 "2017-08-02T14:31:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.