# How to avoid double indexing when using rollover indice **URL:** \ **Category:** Logstash\ **Created:** [September 14, 2022, 5:34am UTC](https://discuss.elastic.co/t/how-to-avoid-double-indexing-when-using-rollover-indice/314366 "2022-09-14T05:34:10Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![PRASHANT\_MEHTA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prashant_mehta/32/101764_2.png) [@PRASHANT\_MEHTA](https://discuss.elastic.co/u/PRASHANT_MEHTA)\ **Post date:** [September 14, 2022, 5:34am UTC](https://discuss.elastic.co/t/how-to-avoid-double-indexing-when-using-rollover-indice/314366/1 "2022-09-14T05:34:10Z")
Hello All, I'm using the rollover feature for my indices on daily basis along with doc\_as\_upsert,to maintain unique documents only.The rollover Index gets deleted after 30 days. I can see the issue of double indexing,Everytime in the newly created index the documents is indexed.Due to this duplications the kibana visulizations shows wrong values. Though I'm aware that through ILM single index can be made and data could be kept for N days and then rollover.But the idea is to maintain only single ILM policy for all indices i.e data should be kept for 1 month(with daily rollover) and then deleted.The challenge here is for all indices showing unique values done through logstash upsert shows duplicate due to rollover. We can't afford duplicate documents on searching. What are the solutions for this issue? Thanx ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 14, 2022, 5:35am UTC](https://discuss.elastic.co/t/how-to-avoid-double-indexing-when-using-rollover-indice/314366/2 "2022-09-14T05:35:12Z")
Why do you have duplicate documents coming in? ---
**Author:** ![PRASHANT\_MEHTA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prashant_mehta/32/101764_2.png) [@PRASHANT\_MEHTA](https://discuss.elastic.co/u/PRASHANT_MEHTA)\ **Post date:** [September 14, 2022, 5:42am UTC](https://discuss.elastic.co/t/how-to-avoid-double-indexing-when-using-rollover-indice/314366/3 "2022-09-14T05:42:40Z")
Hello @Mark_Walkom , Thanx for your quick response! I'm fetching the data related to some of our applications through perl scripts,these scripts run for different time intervals ex:every 5,10,15...mins.So for single day there is no duplication,but next day when index gets rollover the same data is fetched through scripts and hence the duplication. I'm trying to find solution where in previous day index data is not searched for use cases using document\_as\_upsert. Thanx ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 14, 2022, 5:44am UTC](https://discuss.elastic.co/t/how-to-avoid-double-indexing-when-using-rollover-indice/314366/4 "2022-09-14T05:44:31Z")
> [@PRASHANT\_MEHTA](#): > > I'm trying to find solution where in previous day index data is not searched for use cases using > document\_as\_upsert. Sounds like that won't work because the duplicate is being put into a new index, so there's nothing to update. Is there a reason you are not using Elastic Agent/Beats? ---
**Author:** ![PRASHANT\_MEHTA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prashant_mehta/32/101764_2.png) [@PRASHANT\_MEHTA](https://discuss.elastic.co/u/PRASHANT_MEHTA)\ **Post date:** [September 14, 2022, 5:59am UTC](https://discuss.elastic.co/t/how-to-avoid-double-indexing-when-using-rollover-indice/314366/5 "2022-09-14T05:59:43Z")
Yes I'm using beats agents metricbeat for server metrics,filebeat for someusecase. But most of the cases perl generated data is maintained through logstash pipeline due to its rich plugin features. I didn't understood exactly your intention to ask in this case about using elastic agents and would it resolve this issue in anyways? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 14, 2022, 6:07am UTC](https://discuss.elastic.co/t/how-to-avoid-double-indexing-when-using-rollover-indice/314366/6 "2022-09-14T06:07:07Z")
I think that Filebeat should handle the duplicates a little more effectively. ---
**Author:** ![PRASHANT\_MEHTA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prashant_mehta/32/101764_2.png) [@PRASHANT\_MEHTA](https://discuss.elastic.co/u/PRASHANT_MEHTA)\ **Post date:** [September 14, 2022, 6:17am UTC](https://discuss.elastic.co/t/how-to-avoid-double-indexing-when-using-rollover-indice/314366/7 "2022-09-14T06:17:10Z")
I guess filebeat would only be helpful in case of structured log line(log ingest pipeline),here with in scripts the data is at times fetched through database and sometimes data is fetched throug scripts. Due to this logstash is used. How could filebeat could help for duplicates? , Can you please explain a bit. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 14, 2022, 6:18am UTC](https://discuss.elastic.co/t/how-to-avoid-double-indexing-when-using-rollover-indice/314366/8 "2022-09-14T06:18:52Z")
Because it keeps track of what has been read from a file, so it will not usually send the same entry more than once. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 12, 2022, 6:19am UTC](https://discuss.elastic.co/t/how-to-avoid-double-indexing-when-using-rollover-indice/314366/9 "2022-10-12T06:19:26Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.