# Data is duplicating without restart of any services of kibana

**URL:** <https://discuss.elastic.co/t/data-is-duplicating-without-restart-of-any-services-of-kibana/356616>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-monitoring\
**Created:** [April 2, 2024, 1:17pm UTC](https://discuss.elastic.co/t/data-is-duplicating-without-restart-of-any-services-of-kibana/356616 "2024-04-02T13:17:10Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![2328943\_dc](https://avatars.discourse-cdn.com/v4/letter/2/a9adbd/32.png) [@2328943\_dc](https://discuss.elastic.co/u/2328943_dc)\
**Post date:** [April 2, 2024, 1:17pm UTC](https://discuss.elastic.co/t/data-is-duplicating-without-restart-of-any-services-of-kibana/356616/1 "2024-04-02T13:17:10Z")

</div>

for payment URL index hits are coming after every one min for 2 different urls.  
i.e for each url we are getting 60 hits per min.

we have created sincedb path for index and observed data but we found that data is duplicating sometimes without taking restart of kibana services  
As attached in below screenshot we can see sudden rise in the graph.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/2/a28ccf2a9b9c66824177ccda111113a6632339b7.png)

Please find configuration file for index as below.

input {  
file  
{  
path =\> ["/log/AnsibleLogs/Formatfile.txt"]  
tags =\> ["paymenturls"]  
# sincedb\_path =\> "/dev/null"  
# start\_position =\> "end"  
# ignore\_older =\> "3m"  
sincedb\_path =\> "/var/lib/logstash/plugins/inputs/file/.sincedb\_9e5a6659de603250b13193c70cd50ef7"

```
     }

```

}

filter{

if "paymenturls" in [tags]{

grok {  
match =\> { "message" =\> "%{DATA:url}, %{DAY:day} %{MONTH:month} (?[0-9]{2}) %{TIME:time} %{WORD:Zone} %{YEAR:year}, %{NUMBER:Status\_code1}, %{GREEDYDATA:ErrorMessage}" }  
}

mutate {  
add\_field =\> {  
"timestamp" =\> "%{date} %{month} %{year} %{time}"  
}  
}

}

}

output  
{  
if "paymenturls" in [tags]  
{  
elasticsearch  
{  
hosts =\> ["[https://elkmonp4.newindia.co.in:9200](https://elkmonp4.newindia.co.in:9200)"]  
index =\> "niapaymenturls-%{+YYYY.MM.dd}"  
ssl =\> true  
cacert =\> ["/etc/logstash/ELKMONP4.crt"]  
user =\> "elastic"  
password =\> "Password@123"  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [April 2, 2024, 1:24pm UTC](https://discuss.elastic.co/t/data-is-duplicating-without-restart-of-any-services-of-kibana/356616/2 "2024-04-02T13:24:21Z")

</div>

> [@2328943\_dc](#):
>
> /log/AnsibleLogs/Formatfile.txt

Is this a network share or a local path in your server?

Also, is this file rotated or recreated frequently?

---

<div class="post-metadata">

**Author:** ![2328943\_dc](https://avatars.discourse-cdn.com/v4/letter/2/a9adbd/32.png) [@2328943\_dc](https://discuss.elastic.co/u/2328943_dc)\
**Post date:** [April 3, 2024, 6:03am UTC](https://discuss.elastic.co/t/data-is-duplicating-without-restart-of-any-services-of-kibana/356616/3 "2024-04-03T06:03:37Z")

</div>

Not a network shared file  
This file is recreated based on the changes at the source server

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 3, 2024, 6:15am UTC](https://discuss.elastic.co/t/data-is-duplicating-without-restart-of-any-services-of-kibana/356616/4 "2024-04-03T06:15:37Z")

</div>

The sincedb functionality assumes the file is appended to and not rewritten. If a file is rewritten with the same initial content and some new lines it will appear as a new file as the inode will change and the whole file will be reprocessed. This may explain why you are seeing duplicates.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 1, 2024, 6:16am UTC](https://discuss.elastic.co/t/data-is-duplicating-without-restart-of-any-services-of-kibana/356616/5 "2024-05-01T06:16:15Z")

</div>

Closed on May 1, 2024, 6:16am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.