# Duplicate documents found in ES after indexing **URL:** \ **Category:** Elasticsearch\ **Created:** [November 27, 2019, 9:47am UTC](https://discuss.elastic.co/t/duplicate-documents-found-in-es-after-indexing/209647 "2019-11-27T09:47:56Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![bibin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bibin/32/75358_2.png) [@bibin](https://discuss.elastic.co/u/bibin)\ **Post date:** [November 27, 2019, 9:47am UTC](https://discuss.elastic.co/t/duplicate-documents-found-in-es-after-indexing/209647/1 "2019-11-27T09:47:56Z")
Hi friends, I am facing a serious issue of duplicate documents found in ES after indexing... log format // 10.8.1.90,23-11-2019 00:00:07,CURRENT,Broadband,Idea/Vodafone,Failure 10.8.1.90,23-11-2019 01:30:08,CURRENT,Broadband,Idea/Vodafone,Failure 10.8.1.80,23-11-2019 00:00:09,CURRENT,Broadband,Idea/Vodafone,Success 10.8.1.80,23-11-2019 01:30:08,CURRENT,Idea/Vodafone,Broadband,Success 10.8.1.74,23-11-2019 10:06:02,CURRENT,Idea/Vodafone,Broadband,Failure 10.8.1.74,23-11-2019 10:07:02,CURRENT,Idea/Vodafone,Broadband,Failure 10.8.1.74,23-11-2019 10:08:02,CURRENT,Idea/Vodafone,Broadband,Failure // and I am using the following configuration file for indexing... // input { file { **path =\> "/home/logs/connection.csv"** max\_open\_files =\> 17000 start\_position =\> "beginning" sincedb\_path =\> "/home/ALL/since\_connection\_november.db" } } filter { #10.8.0.68,10-11-2019 12:00:12,CURRENT,Broadband,Idea/Vodafone,Success #10.8.0.68,10-11-2019 13:30:14,CURRENT,Idea/Vodafone,Broadband,Failure dissect { mapping =\> { "message" =\> "%{ip},%{happened\_instant},%{time\_slot},%{provider\_from},%{provider\_to},%{status}" } } date { match =\> ["happened\_instant", "dd-MM-yyyy HH:mm:ss"] **target =\> "failover\_date"** } } output { elasticsearch { hosts =\> "server\_domain:9292" index =\> "connection\_november" } stdout {} } // I am using the **failover\_date** date target as timestamp while indexing. My file line count is almost 2744744 records but after indexing and checking **discovery** found that its showing hits of almost 4 times more ie 8745544 records. When I checked for the duplicate data, found that each document is inserted multiple times with difference only in **@timestamp**. **ie same document inserted multiple times at different time..** Please find the format of two same docs at with different timestamp which I found while expanding the docs in discover tab... first doc //////// { "\_index": "connection\_november", "\_type": "\_doc", "\_id": "aEEdn24BNAK6JMz4dK\_F", "\_version": 1, "\_score": null, "\_source": { "ip": "10.8.1.90", **"failover\_date": "2019-11-22T18:30:07.000Z"** , "host": "swupdate", "provider\_from": "Broadband", **"message": "10.8.1.90,23-11-2019 00:00:07,CURRENT,Broadband,Idea/Vodafone,Failure",** **"@timestamp": "2019-11-24T20:32:17.903Z",** "@version": "1", "status": "Failure", "happened\_instant": "23-11-2019 00:00:07", "path": "/home/logs/connection.csv", "time\_slot": "CURRENT", "provider\_to": "Idea/Vodafone" }, "fields": { "failover\_date": [ **"2019-11-22T18:30:07.000Z"** ], "@timestamp": [ **"2019-11-24T20:32:17.903Z"** ] }, "sort": [ 1574627537903 ] } //// second doc //// { "\_index": "connection\_november", "\_type": "\_doc", "\_id": "2vH3mW4BNAK6JMz4Btpt", "\_version": 1, "\_score": null, "\_source": { "ip": "10.8.1.90", **"failover\_date": "2019-11-22T18:30:07.000Z",** "host": "swupdate", "provider\_from": "Broadband", **"message": "10.8.1.90,23-11-2019 00:00:07,CURRENT,Broadband,Idea/Vodafone,Failure",** **"@timestamp": "2019-11-23T20:32:13.228Z",** "@version": "1", "status": "Failure", "happened\_instant": "23-11-2019 00:00:07", "path": "/home/logs/connection.csv", "time\_slot": "CURRENT", "provider\_to": "Idea/Vodafone" }, "fields": { "failover\_date": [ **"2019-11-22T18:30:07.000Z"** ], "@timestamp": [ **"2019-11-23T20:32:13.228Z"** ] }, "sort": [ 1574541133228 ] } /// Please note that the bolded portion is the same one with different timestamb. Can anyone suggest me why this issue occurs...?,or can suggest any solution... Please note that I am new to ES.. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 27, 2019, 12:07pm UTC](https://discuss.elastic.co/t/duplicate-documents-found-in-es-after-indexing/209647/2 "2019-11-27T12:07:59Z")
How is data added to the file you are reading from? ---
**Author:** ![bibin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bibin/32/75358_2.png) [@bibin](https://discuss.elastic.co/u/bibin)\ **Post date:** [November 27, 2019, 12:13pm UTC](https://discuss.elastic.co/t/duplicate-documents-found-in-es-after-indexing/209647/3 "2019-11-27T12:13:19Z")
Hi Christian, Its been added simultaneously from different edge devices like beaglebone.. Its been appended by almost 250 devices .. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 27, 2019, 1:14pm UTC](https://discuss.elastic.co/t/duplicate-documents-found-in-es-after-indexing/209647/4 "2019-11-27T13:14:21Z")
Is this a local file being appended to? Does inode change when updated? ---
**Author:** ![bibin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bibin/32/75358_2.png) [@bibin](https://discuss.elastic.co/u/bibin)\ **Post date:** [November 28, 2019, 6:10am UTC](https://discuss.elastic.co/t/duplicate-documents-found-in-es-after-indexing/209647/5 "2019-11-28T06:10:41Z")
Is this a local file being appended to? **yes**.It is a local file in the server in which logstash is running.but it is updated by many remote edge devices via internet every 5mins. ``` Did you mean .. whether my logstash and ES is running in same server..? NO both are in separate servers. ``` Does inode change when updated? ``` yes, it will change inode of the server in which logstash is running,because there are many other files which are transfered simultaneously to this server. ``` inode count is 67033. ---
**Author:** ![bibin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bibin/32/75358_2.png) [@bibin](https://discuss.elastic.co/u/bibin)\ **Post date:** [December 5, 2019, 8:25am UTC](https://discuss.elastic.co/t/duplicate-documents-found-in-es-after-indexing/209647/6 "2019-12-05T08:25:58Z")
Hi Christain, Can you help me with this issue...? Does the above provided details regarding inode is sufficient..? Thanks and regards ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 5, 2019, 8:39am UTC](https://discuss.elastic.co/t/duplicate-documents-found-in-es-after-indexing/209647/7 "2019-12-05T08:39:43Z")
Does the inode of the file (not inode count) change as it is being updated? ---
**Author:** ![bibin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bibin/32/75358_2.png) [@bibin](https://discuss.elastic.co/u/bibin)\ **Post date:** [December 9, 2019, 7:49am UTC](https://discuss.elastic.co/t/duplicate-documents-found-in-es-after-indexing/209647/8 "2019-12-09T07:49:22Z")
Yes,christian\_D Its being changed on time. why the inode is changing because the main input file is changed frequently by edge devices simultaneously. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 9, 2019, 8:25am UTC](https://discuss.elastic.co/t/duplicate-documents-found-in-es-after-indexing/209647/9 "2019-12-09T08:25:02Z")
If it changes the file will look like a new file and be reprocessed. It is exoected that data is appended and the inode does not change. ---
**Author:** ![bibin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bibin/32/75358_2.png) [@bibin](https://discuss.elastic.co/u/bibin)\ **Post date:** [December 12, 2019, 5:25pm UTC](https://discuss.elastic.co/t/duplicate-documents-found-in-es-after-indexing/209647/10 "2019-12-12T17:25:06Z")
Hi Christian , Thanks for the help you have rendered so far.I understood the cause of my issue.As you said its because of the change of inode that ES considers my input file as a new file every time its indexing and so indexes it multiple time and atlast resulting in duplicate documents. ---
**Author:** ![bibin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bibin/32/75358_2.png) [@bibin](https://discuss.elastic.co/u/bibin)\ **Post date:** [December 12, 2019, 5:29pm UTC](https://discuss.elastic.co/t/duplicate-documents-found-in-es-after-indexing/209647/11 "2019-12-12T17:29:51Z")
So can you suggest me a solution how to handle my input file which's inode is constanstly changing...? My input file is a concatination of multiple small files from edge devices which is appended to server all time... Actually it uses linux **rsync** to append small files from multiple edge devices to the server.. so that my input file grows every 5 mins ... ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 12, 2019, 8:53pm UTC](https://discuss.elastic.co/t/duplicate-documents-found-in-es-after-indexing/209647/12 "2019-12-12T20:53:46Z")
Can you install Filebeat on the edge devices? ---
**Author:** ![bibin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bibin/32/75358_2.png) [@bibin](https://discuss.elastic.co/u/bibin)\ **Post date:** [December 16, 2019, 4:17pm UTC](https://discuss.elastic.co/t/duplicate-documents-found-in-es-after-indexing/209647/13 "2019-12-16T16:17:22Z")
Its almost 350 devices... Is there any other way...? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 13, 2020, 4:17pm UTC](https://discuss.elastic.co/t/duplicate-documents-found-in-es-after-indexing/209647/14 "2020-01-13T16:17:23Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.