# How to avoid duplication of data while indexing
**URL:** https://discuss.elastic.co/t/how-to-avoid-duplication-of-data-while-indexing/9878
**Category:** Elasticsearch
**Created:** [November 29, 2012, 9:12am UTC](https://discuss.elastic.co/t/how-to-avoid-duplication-of-data-while-indexing/9878 "2012-11-29T09:12:28Z")
**Posts on this page:** 10
**Page:** 1
### Author:  [@divyanshu\_das](https://discuss.elastic.co/u/divyanshu_das)
#### Post date: [November 29, 2012, 9:12am UTC](https://discuss.elastic.co/t/how-to-avoid-duplication-of-data-while-indexing/9878/1 "2012-11-29T09:12:28Z")
I need to index data of 4 million users in elasticsearch using JAVA API.
for each object these are the fields that are getting indexed:-
Name -
Address -
I need to put a check at the time of indexing. If another json object with
the same name as this object has already been indexed, then either discard
this object or overwrite the previous one.
Can someone help me out here?
--
---
### Author:  [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)
#### Post date: [November 29, 2012, 9:28am UTC](https://discuss.elastic.co/t/how-to-avoid-duplication-of-data-while-indexing/9878/2 "2012-11-29T09:28:39Z")
Hello,
You can use the name as an ID:
> **[Elastic — The Search AI Company](https://www.elastic.co)**
>
> Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ...
Then, if you want to overwrite, you can simply index your documents, and
you'll see in the version field how many times you got that overwritten.
Some more information about versioning here:
> **[Elastic — The Search AI Company](https://www.elastic.co)**
>
> Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ...
If you want to discard the new object, you can specify op\_type=create while
indexing:
> **[Elastic — The Search AI Company](https://www.elastic.co)**
>
> Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ...
## Best regards, Radu
[http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene
On Thu, Nov 29, 2012 at 11:12 AM, divyanshu das [divyanshu.das@gmail.com](mailto:divyanshu.das@gmail.com)wrote:
> I need to index data of 4 million users in elasticsearch using JAVA API.
>
> for each object these are the fields that are getting indexed:-
>
> Name -
> Address -
>
> I need to put a check at the time of indexing. If another json object with
> the same name as this object has already been indexed, then either discard
> this object or overwrite the previous one.
>
> Can someone help me out here?
--
---
### Author:  [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)
#### Post date: [November 29, 2012, 9:38am UTC](https://discuss.elastic.co/t/how-to-avoid-duplication-of-data-while-indexing/9878/3 "2012-11-29T09:38:02Z")
You can use a hash of both as the document ID
On Thu, Nov 29, 2012 at 11:12 AM, divyanshu das [divyanshu.das@gmail.com](mailto:divyanshu.das@gmail.com)wrote:
> I need to index data of 4 million users in elasticsearch using JAVA API.
>
> for each object these are the fields that are getting indexed:-
>
> Name -
> Address -
>
> I need to put a check at the time of indexing. If another json object with
> the same name as this object has already been indexed, then either discard
> this object or overwrite the previous one.
>
> Can someone help me out here?
>
> --
--
---
### Author:  [@divyanshu\_das](https://discuss.elastic.co/u/divyanshu_das)
#### Post date: [November 29, 2012, 9:53am UTC](https://discuss.elastic.co/t/how-to-avoid-duplication-of-data-while-indexing/9878/4 "2012-11-29T09:53:32Z")
I am new to this mapping concept on how and where to define it.
I am creating the index like this, met.xb where xb is the json builder,
contains the fields to be indexed.
IndexResponse response = client.prepareIndex("nod",
"rel").setSource(met.xb).execute().actionGet();
I dont see any example in Client API regarding mapping.
How do I make met.xb.field("name", name) as the default ID?
On Thursday, 29 November 2012 15:08:02 UTC+5:30, Itamar Syn-Hershko wrote:
> You can use a hash of both as the document ID
>
> On Thu, Nov 29, 2012 at 11:12 AM, divyanshu das \<[divyan...@gmail.com](mailto:divyan...@gmail.com)\
>
> > wrote:
>
> > I need to index data of 4 million users in elasticsearch using JAVA API.
> >
> > for each object these are the fields that are getting indexed:-
> >
> > Name -
> > Address -
> >
> > I need to put a check at the time of indexing. If another json object
> > with the same name as this object has already been indexed, then either
> > discard this object or overwrite the previous one.
> >
> > Can someone help me out here?
> >
> > --
--
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [November 29, 2012, 10:16am UTC](https://discuss.elastic.co/t/how-to-avoid-duplication-of-data-while-indexing/9878/5 "2012-11-29T10:16:14Z")
Use something like
> IndexResponse response = client.prepareIndex("nod", "rel",
> "id").setSource(met.xb).execute().actionGet();
where id is your name (or best as suggested by Itamar, a HASH of your name).
HTH
David.
Le 29 novembre 2012 à 10:53, divyanshu das [divyanshu.das@gmail.com](mailto:divyanshu.das@gmail.com) a écrit :
> I am new to this mapping concept on how and where to define it.
>
> I am creating the index like this, met.xb where xb is the json builder,
> contains the fields to be indexed.
>
> IndexResponse response = client.prepareIndex("nod",
> "rel").setSource(met.xb).execute().actionGet();
>
> I dont see any example in Client API regarding mapping.
>
> How do I make met.xb.field("name", name) as the default ID?
>
> On Thursday, 29 November 2012 15:08:02 UTC+5:30, Itamar Syn-Hershko wrote:
>
> > > You can use a hash of both as the document ID
> >
> > On Thu, Nov 29, 2012 at 11:12 AM, divyanshu das [divyan...@gmail.com](mailto:divyan...@gmail.com)
> > wrote:
> > \> \> \> I need to index data of 4 million users in elasticsearch using
> > \> \> \> JAVA API.
> >
> > > ```
> > > for each object these are the fields that are getting indexed:-
> > >
> > > Name -
> > > Address -
> > >
> > > I need to put a check at the time of indexing. If another json object
> > >
> > > ```
> > >
> > > with the same name as this object has already been indexed, then either
> > > discard this object or overwrite the previous one.
> > >
> > > ```
> > > Can someone help me out here?
> > >
> > > --
> > >
> > > ```
> > >
> > > > > >
>
> --
--
David Pilato
[http://www.scrutmydocs.org/](http://www.scrutmydocs.org/)
[http://dev.david.pilato.fr/](http://dev.david.pilato.fr/)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
--
---
### Author:  [@divyanshu\_das](https://discuss.elastic.co/u/divyanshu_das)
#### Post date: [November 29, 2012, 10:49am UTC](https://discuss.elastic.co/t/how-to-avoid-duplication-of-data-while-indexing/9878/6 "2012-11-29T10:49:03Z")
@David,
so, lets say i define a new string,
String newid = abc.getName() // abc.getName() returns the name
IndexResponse response = client.prepareIndex("nod", "rel",
newid).setSource(met.xb).execute().actionGet();
Will this work?
On Thursday, 29 November 2012 15:46:14 UTC+5:30, David Pilato wrote:
> Use something like
>
> IndexResponse response = client.prepareIndex("nod",
> "rel").setSource(met.xb).execute().actionGet();
>
> where id is your name (or best as suggested by Itamar, a HASH of your
> name).
>
> HTH
> David.
>
> Le 29 novembre 2012 à 10:53, divyanshu das \<[divyan...@gmail.com](mailto:divyan...@gmail.com)\\>
> a écrit :
>
> I am new to this mapping concept on how and where to define it.
>
> I am creating the index like this, met.xb where xb is the json builder,
> contains the fields to be indexed.
>
> IndexResponse response = client.prepareIndex("nod",
> "rel").setSource(met.xb).execute().actionGet();
>
> I dont see any example in Client API regarding mapping.
>
> How do I make met.xb.field("name", name) as the default ID?
>
> On Thursday, 29 November 2012 15:08:02 UTC+5:30, Itamar Syn-Hershko wrote:
>
> You can use a hash of both as the document ID
>
> On Thu, Nov 29, 2012 at 11:12 AM, divyanshu das [divyan...@gmail.com](mailto:divyan...@gmail.com)wrote:
>
> I need to index data of 4 million users in elasticsearch using JAVA API.
>
> for each object these are the fields that are getting indexed:-
>
> Name -
> Address -
>
> I need to put a check at the time of indexing. If another json object
> with the same name as this object has already been indexed, then either
> discard this object or overwrite the previous one.
>
> Can someone help me out here?
>
> --
>
> --
>
> --
> David Pilato
> [http://www.scrutmydocs.org/](http://www.scrutmydocs.org/)
> [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/)
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
--
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [November 29, 2012, 11:05am UTC](https://discuss.elastic.co/t/how-to-avoid-duplication-of-data-while-indexing/9878/7 "2012-11-29T11:05:53Z")
If I understand what you are after (say index with a given ID): yes.
But, if name is a string like "Hello I'm happy to meet you!", It will not work.
So you have to encode your ID.
To encode it, you can use something like :
>
Or perhaps : org.elasticsearch.common.UUID.fromString(name) will help;
David
Le 29 novembre 2012 à 11:49, divyanshu das [divyanshu.das@gmail.com](mailto:divyanshu.das@gmail.com) a écrit :
> @David,
>
> so, lets say i define a new string,
>
> String newid = abc.getName() // abc.getName() returns the name
>
> IndexResponse response = client.prepareIndex("nod", "rel",
> newid).setSource(met.xb).execute().actionGet();
>
> Will this work?
>
> On Thursday, 29 November 2012 15:46:14 UTC+5:30, David Pilato wrote:
>
> > >
> >
> > Use something like
> >
> > ```
> > > > >
> >
> > ```
> >
> > > > > IndexResponse response = client.prepareIndex("nod",
> > > > > "rel").setSource(met.xb).execute().actionGet();
> >
> > where id is your name (or best as suggested by Itamar, a HASH of your
> > name).
> >
> > HTH
> > David.
> >
> > Le 29 novembre 2012 à 10:53, divyanshu das \< [divyan...@gmail.com](mailto:divyan...@gmail.com)\> a écrit
> > :
> >
> > ```
> > > > > I am new to this mapping concept on how and where to define it.
> >
> > ```
> >
> > > ```
> > > I am creating the index like this, met.xb where xb is the json
> > >
> > > ```
> > >
> > > builder, contains the fields to be indexed.
> > >
> > > ```
> > > IndexResponse response = client.prepareIndex("nod",
> > >
> > > ```
> > >
> > > "rel").setSource(met.xb).execute().actionGet();
> > >
> > > ```
> > > I dont see any example in Client API regarding mapping.
> > >
> > > How do I make met.xb.field("name", name) as the default ID?
> > >
> > > On Thursday, 29 November 2012 15:08:02 UTC+5:30, Itamar Syn-Hershko
> > >
> > > ```
> > >
> > > wrote:
> > > \> \> \> \> You can use a hash of both as the document ID
> > >
> > > > ```
> > > > On Thu, Nov 29, 2012 at 11:12 AM, divyanshu das
> > > >
> > > > ```
> > > >
> > > > [divyan...@gmail.com](mailto:divyan...@gmail.com) wrote:
> > > > \> \> \> \> \> I need to index data of 4 million users in
> > > > \> \> \> \> \> elasticsearch using JAVA API.
> > > >
> > > > > ```
> > > > > for each object these are the fields that are getting
> > > > >
> > > > > ```
> > > > >
> > > > > indexed:-
> > > > >
> > > > > ```
> > > > > Name -
> > > > > Address -
> > > > >
> > > > > I need to put a check at the time of indexing. If another json
> > > > >
> > > > > ```
> > > > >
> > > > > object with the same name as this object has already been indexed,
> > > > > then either discard this object or overwrite the previous one.
> > > > >
> > > > > ```
> > > > > Can someone help me out here?
> > > > >
> > > > > --
> > > > >
> > > > > > > > > > > >
> > > > >
> > > > > ```
> > >
> > > ```
> > > --
> > >
> > > ```
> > >
> > > > >
> >
> > --
> > David Pilato
> > [http://www.scrutmydocs.org/](http://www.scrutmydocs.org/) [http://www.scrutmydocs.org/](http://www.scrutmydocs.org/)
> > [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/)
> > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> >
> > >
>
> --
--
David Pilato
[http://www.scrutmydocs.org/](http://www.scrutmydocs.org/)
[http://dev.david.pilato.fr/](http://dev.david.pilato.fr/)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
--
---
### Author:  [@divyanshu\_das](https://discuss.elastic.co/u/divyanshu_das)
#### Post date: [November 29, 2012, 11:39am UTC](https://discuss.elastic.co/t/how-to-avoid-duplication-of-data-while-indexing/9878/8 "2012-11-29T11:39:40Z")
yes, I want to index with a given ID.
So, if the name is one single word "xyz", then it should work.
or if lets say I replace name with a particular ID (not Elasticsearch id
but my own id).
String newid = abc.getID();
IndexResponse response = client.prepareIndex("nod", "rel",
newid).setSource(met.xb).execute().actionGet();
Will this work?
On Thu, Nov 29, 2012 at 4:35 PM, David Pilato [david@pilato.fr](mailto:david@pilato.fr) wrote:
> \*\*
> If I understand what you are after (say index with a given ID): yes.
> But, if name is a string like "Hello I'm happy to meet you!", It will not
> work.
> So you have to encode your ID.
>
> To encode it, you can use something like :
> [https://github.com/scrutmydocs/scrutmydocs/blob/master/src/main/java/org/scrutmydocs/webapp/util/SignTool.java](https://github.com/scrutmydocs/scrutmydocs/blob/master/src/main/java/org/scrutmydocs/webapp/util/SignTool.java)
>
> Or perhaps : org.elasticsearch.common.UUID.fromString(name) will help;
>
> David
>
> Le 29 novembre 2012 à 11:49, divyanshu das [divyanshu.das@gmail.com](mailto:divyanshu.das@gmail.com) a
> écrit :
>
> @David,
>
> so, lets say i define a new string,
>
> String newid = abc.getName() // abc.getName() returns the name
>
> IndexResponse response = client.prepareIndex("nod", "rel",
> newid).setSource(met.xb).execute().actionGet();
>
> Will this work?
>
> On Thursday, 29 November 2012 15:46:14 UTC+5:30, David Pilato wrote:
>
> Use something like
>
> IndexResponse response = client.prepareIndex("nod",
> "rel").setSource(met.xb).execute().actionGet();
>
> where id is your name (or best as suggested by Itamar, a HASH of your
> name).
>
> HTH
> David.
>
> Le 29 novembre 2012 à 10:53, divyanshu das \< [divyan...@gmail.com](mailto:divyan...@gmail.com)\> a
> écrit :
>
> I am new to this mapping concept on how and where to define it.
>
> I am creating the index like this, met.xb where xb is the json builder,
> contains the fields to be indexed.
>
> IndexResponse response = client.prepareIndex("nod",
> "rel").setSource(met.xb).execute().actionGet();
>
> I dont see any example in Client API regarding mapping.
>
> How do I make met.xb.field("name", name) as the default ID?
>
> On Thursday, 29 November 2012 15:08:02 UTC+5:30, Itamar Syn-Hershko wrote:
>
> You can use a hash of both as the document ID
>
> On Thu, Nov 29, 2012 at 11:12 AM, divyanshu das [divyan...@gmail.com](mailto:divyan...@gmail.com)wrote:
>
> I need to index data of 4 million users in elasticsearch using JAVA API.
>
> for each object these are the fields that are getting indexed:-
>
> Name -
> Address -
>
> I need to put a check at the time of indexing. If another json object
> with the same name as this object has already been indexed, then either
> discard this object or overwrite the previous one.
>
> Can someone help me out here?
>
> --
>
> --
>
> --
> David Pilato
> [http://www.scrutmydocs.org/](http://www.scrutmydocs.org/)
> [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/)
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>
> --
>
> --
> David Pilato
> [http://www.scrutmydocs.org/](http://www.scrutmydocs.org/)
> [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/)
> Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
>
> --
--
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [November 29, 2012, 2:47pm UTC](https://discuss.elastic.co/t/how-to-avoid-duplication-of-data-while-indexing/9878/9 "2012-11-29T14:47:22Z")
Yes...
I would say: "just test it!" It's so easy to start an ES node in Java.
Have a look at: [GitHub - elasticsearchfr/hands-on: Hands On Lab](https://github.com/elasticsearchfr/hands-on)
It will help you to start in Java with ES.
HTH
David.
Le 29 novembre 2012 à 12:39, divyanshu das [divyanshu.das@gmail.com](mailto:divyanshu.das@gmail.com) a écrit :
> yes, I want to index with a given ID.
>
> So, if the name is one single word "xyz", then it should work.
>
> or if lets say I replace name with a particular ID (not Elasticsearch id but
> my own id).
>
> String newid = abc.getID();
>
> IndexResponse response = client.prepareIndex("nod", "rel",
> newid).setSource(met.xb). execute().actionGet();
>
> Will this work?
>
> On Thu, Nov 29, 2012 at 4:35 PM, David Pilato \<[david@pilato.fr](mailto:david@pilato.fr)
> [mailto:david@pilato.fr](mailto:david@pilato.fr) \> wrote:
>
> > > If I understand what you are after (say index with a given ID): yes.
> > > But, if name is a string like "Hello I'm happy to meet you!", It will not
> > > work.
> > > So you have to encode your ID.
> >
> > To encode it, you can use something like :
> > [https://github.com/scrutmydocs/scrutmydocs/blob/master/src/main/java/org/scrutmydocs/webapp/util/SignTool.java](https://github.com/scrutmydocs/scrutmydocs/blob/master/src/main/java/org/scrutmydocs/webapp/util/SignTool.java)
> > [https://github.com/scrutmydocs/scrutmydocs/blob/master/src/main/java/org/scrutmydocs/webapp/util/SignTool.java](https://github.com/scrutmydocs/scrutmydocs/blob/master/src/main/java/org/scrutmydocs/webapp/util/SignTool.java)
> >
> > Or perhaps : org.elasticsearch.common.UUID.fromString(name) will help;
> >
> > David
> >
> > Le 29 novembre 2012 à 11:49, divyanshu das \< [divyanshu.das@gmail.com](mailto:divyanshu.das@gmail.com)
> > [mailto:divyanshu.das@gmail.com](mailto:divyanshu.das@gmail.com) \> a écrit :
> >
> > ```
> > > > > @David,
> >
> > ```
> >
> > > ```
> > > so, lets say i define a new string,
> > >
> > > String newid = abc.getName() // abc.getName() returns the name
> > >
> > > IndexResponse response = client.prepareIndex("nod", "rel",
> > >
> > > ```
> > >
> > > newid).setSource(met.xb).execute().actionGet();
> > >
> > > ```
> > > Will this work?
> > >
> > > On Thursday, 29 November 2012 15:46:14 UTC+5:30, David Pilato wrote:
> > > > > > >
> > >
> > > ```
> > >
> > > > ```
> > > > Use something like
> > > >
> > > > > > > > >
> > > >
> > > > ```
> > > >
> > > > > ```
> > > > > > > > > IndexResponse response = client.prepareIndex("nod",
> > > > > > > > > "rel").setSource(met.xb).execute().actionGet();
> > > > >
> > > > > ```
> > > >
> > > > ```
> > > > where id is your name (or best as suggested by Itamar, a HASH of
> > > >
> > > > ```
> > > >
> > > > your name).
> > > >
> > > > ```
> > > > HTH
> > > > David.
> > > >
> > > > Le 29 novembre 2012 à 10:53, divyanshu das < divyan...@gmail.com>
> > > >
> > > > ```
> > > >
> > > > a écrit :
> > > >
> > > > ```
> > > > > > > > > I am new to this mapping concept on how and where to
> > > > > > > > > define it.
> > > >
> > > > ```
> > > >
> > > > > ```
> > > > > I am creating the index like this, met.xb where xb is the json
> > > > >
> > > > > ```
> > > > >
> > > > > builder, contains the fields to be indexed.
> > > > >
> > > > > ```
> > > > > IndexResponse response = client.prepareIndex("nod",
> > > > >
> > > > > ```
> > > > >
> > > > > "rel").setSource(met.xb).execute().actionGet();
> > > > >
> > > > > ```
> > > > > I dont see any example in Client API regarding mapping.
> > > > >
> > > > > How do I make met.xb.field("name", name) as the default ID?
> > > > >
> > > > > On Thursday, 29 November 2012 15:08:02 UTC+5:30, Itamar
> > > > >
> > > > > ```
> > > > >
> > > > > Syn-Hershko wrote:
> > > > > \> \> \> \> \> \> You can use a hash of both as the
> > > > > \> \> \> \> \> \> document ID
> > > > >
> > > > > > ```
> > > > > > On Thu, Nov 29, 2012 at 11:12 AM, divyanshu das
> > > > > >
> > > > > > ```
> > > > > >
> > > > > > [divyan...@gmail.com](mailto:divyan...@gmail.com) wrote:
> > > > > > \> \> \> \> \> \> \> I need to index data of 4 million users in
> > > > > > \> \> \> \> \> \> \> elasticsearch using JAVA API.
> > > > > >
> > > > > > > ```
> > > > > > > for each object these are the fields that are getting
> > > > > > >
> > > > > > > ```
> > > > > > >
> > > > > > > indexed:-
> > > > > > >
> > > > > > > ```
> > > > > > > Name -
> > > > > > > Address -
> > > > > > >
> > > > > > > I need to put a check at the time of indexing. If
> > > > > > >
> > > > > > > ```
> > > > > > >
> > > > > > > another json object with the same name as this object has already
> > > > > > > been indexed, then either discard this object or overwrite the
> > > > > > > previous one.
> > > > > > >
> > > > > > > ```
> > > > > > > Can someone help me out here?
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > >
> > > > > > > ```
> > > > >
> > > > > ```
> > > > > --
> > > > >
> > > > > > > > >
> > > > >
> > > > > ```
> > > >
> > > > ```
> > > > --
> > > > David Pilato
> > > > http://www.scrutmydocs.org/
> > > > http://dev.david.pilato.fr/
> > > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> > > > > > >
> > > >
> > > > ```
> > >
> > > ```
> > > --
> > >
> > > ```
> > >
> > > > >
> >
> > --
> > David Pilato
> > [http://www.scrutmydocs.org/](http://www.scrutmydocs.org/) [http://www.scrutmydocs.org/](http://www.scrutmydocs.org/)
> > [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/)
> > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> >
> > --
> >
> > >
>
> --
--
David Pilato
[http://www.scrutmydocs.org/](http://www.scrutmydocs.org/)
[http://dev.david.pilato.fr/](http://dev.david.pilato.fr/)
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
--
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 3:02am UTC](https://discuss.elastic.co/t/how-to-avoid-duplication-of-data-while-indexing/9878/10 "2017-07-06T03:02:10Z")