How to avoid duplication of data while indexing

divyanshu_das · November 29, 2012, 9:12am

I need to index data of 4 million users in elasticsearch using JAVA API.

for each object these are the fields that are getting indexed:-

Name -
Address -

I need to put a check at the time of indexing. If another json object with
the same name as this object has already been indexed, then either discard
this object or overwrite the previous one.

Can someone help me out here?

--

radu_gheorghe · November 29, 2012, 9:28am

Hello,

You can use the name as an ID:

Then, if you want to overwrite, you can simply index your documents, and
you'll see in the version field how many times you got that overwritten.
Some more information about versioning here:

If you want to discard the new object, you can specify op_type=create while
indexing:

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

On Thu, Nov 29, 2012 at 11:12 AM, divyanshu das divyanshu.das@gmail.comwrote:

I need to index data of 4 million users in elasticsearch using JAVA API.

for each object these are the fields that are getting indexed:-

Name -
Address -

I need to put a check at the time of indexing. If another json object with
the same name as this object has already been indexed, then either discard
this object or overwrite the previous one.

Can someone help me out here?

--

Itamar_Syn_Hershko · November 29, 2012, 9:38am

You can use a hash of both as the document ID

On Thu, Nov 29, 2012 at 11:12 AM, divyanshu das divyanshu.das@gmail.comwrote:

I need to index data of 4 million users in elasticsearch using JAVA API.

for each object these are the fields that are getting indexed:-

Name -
Address -

I need to put a check at the time of indexing. If another json object with
the same name as this object has already been indexed, then either discard
this object or overwrite the previous one.

Can someone help me out here?

--

--

divyanshu_das · November 29, 2012, 9:53am

I am new to this mapping concept on how and where to define it.

I am creating the index like this, met.xb where xb is the json builder,
contains the fields to be indexed.

IndexResponse response = client.prepareIndex("nod",
"rel").setSource(met.xb).execute().actionGet();

I dont see any example in Client API regarding mapping.

How do I make met.xb.field("name", name) as the default ID?

On Thursday, 29 November 2012 15:08:02 UTC+5:30, Itamar Syn-Hershko wrote:

You can use a hash of both as the document ID

On Thu, Nov 29, 2012 at 11:12 AM, divyanshu das <divyan...@gmail.com<javascript:>

wrote:

I need to index data of 4 million users in elasticsearch using JAVA API.

for each object these are the fields that are getting indexed:-

Name -
Address -

I need to put a check at the time of indexing. If another json object
with the same name as this object has already been indexed, then either
discard this object or overwrite the previous one.

Can someone help me out here?

--

--

dadoonet · November 29, 2012, 10:16am

Use something like

IndexResponse response = client.prepareIndex("nod", "rel",
"id").setSource(met.xb).execute().actionGet();

where id is your name (or best as suggested by Itamar, a HASH of your name).

HTH
David.

Le 29 novembre 2012 à 10:53, divyanshu das divyanshu.das@gmail.com a écrit :

I am new to this mapping concept on how and where to define it.

I am creating the index like this, met.xb where xb is the json builder,
contains the fields to be indexed.

IndexResponse response = client.prepareIndex("nod",
"rel").setSource(met.xb).execute().actionGet();

I dont see any example in Client API regarding mapping.

How do I make met.xb.field("name", name) as the default ID?

On Thursday, 29 November 2012 15:08:02 UTC+5:30, Itamar Syn-Hershko wrote:
You can use a hash of both as the document ID

On Thu, Nov 29, 2012 at 11:12 AM, divyanshu das divyan...@gmail.com
wrote:
> > > I need to index data of 4 million users in elasticsearch using
> > > JAVA API.
 for each object these are the fields that are getting indexed:-

 Name -
 Address -

 I need to put a check at the time of indexing. If another json object
with the same name as this object has already been indexed, then either
discard this object or overwrite the previous one.
 Can someone help me out here?




 --
--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

divyanshu_das · November 29, 2012, 10:49am

@David,

so, lets say i define a new string,

String newid = abc.getName() // abc.getName() returns the name

IndexResponse response = client.prepareIndex("nod", "rel",
newid).setSource(met.xb).execute().actionGet();

Will this work?

On Thursday, 29 November 2012 15:46:14 UTC+5:30, David Pilato wrote:

Use something like

IndexResponse response = client.prepareIndex("nod",
"rel").setSource(met.xb).execute().actionGet();

where id is your name (or best as suggested by Itamar, a HASH of your
name).

HTH
David.

Le 29 novembre 2012 à 10:53, divyanshu das <divyan...@gmail.com<javascript:>>
a écrit :

I am new to this mapping concept on how and where to define it.

I am creating the index like this, met.xb where xb is the json builder,
contains the fields to be indexed.

IndexResponse response = client.prepareIndex("nod",
"rel").setSource(met.xb).execute().actionGet();

I dont see any example in Client API regarding mapping.

How do I make met.xb.field("name", name) as the default ID?

On Thursday, 29 November 2012 15:08:02 UTC+5:30, Itamar Syn-Hershko wrote:

You can use a hash of both as the document ID

On Thu, Nov 29, 2012 at 11:12 AM, divyanshu das divyan...@gmail.comwrote:

I need to index data of 4 million users in elasticsearch using JAVA API.

for each object these are the fields that are getting indexed:-

Name -
Address -

I need to put a check at the time of indexing. If another json object
with the same name as this object has already been indexed, then either
discard this object or overwrite the previous one.

Can someone help me out here?

--

--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

dadoonet · November 29, 2012, 11:05am

If I understand what you are after (say index with a given ID): yes.
But, if name is a string like "Hello I'm happy to meet you!", It will not work.
So you have to encode your ID.

To encode it, you can use something like :

github.com

scrutmydocs/scrutmydocs/blob/master/src/main/java/org/scrutmydocs/webapp/util/SignTool.java

/*
 * Licensed to scrutmydocs.org (the "Author") under one
 * or more contributor license agreements.  See the NOTICE file
 * distributed with this work for additional information
 * regarding copyright ownership. Author licenses this
 * file to you under the Apache License, Version 2.0 (the
 * "License"); you may not use this file except in compliance
 * with the License.  You may obtain a copy of the License at
 *
 *    http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing,
 * software distributed under the License is distributed on an
 * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
 * KIND, either express or implied.  See the License for the
 * specific language governing permissions and limitations
 * under the License.
 */

package org.scrutmydocs.webapp.util;

This file has been truncated. show original

Or perhaps : org.elasticsearch.common.UUID.fromString(name) will help;

David

Le 29 novembre 2012 à 11:49, divyanshu das divyanshu.das@gmail.com a écrit :

@David,

so, lets say i define a new string,

String newid = abc.getName() // abc.getName() returns the name

IndexResponse response = client.prepareIndex("nod", "rel",
newid).setSource(met.xb).execute().actionGet();

Will this work?

On Thursday, 29 November 2012 15:46:14 UTC+5:30, David Pilato wrote:
Use something like
   > > > 
IndexResponse response = client.prepareIndex("nod",
"rel").setSource(met.xb).execute().actionGet();

where id is your name (or best as suggested by Itamar, a HASH of your
name).

HTH
David.

Le 29 novembre 2012 à 10:53, divyanshu das < divyan...@gmail.com> a écrit
:
> > > I am new to this mapping concept on how and where to define it.
I am creating the index like this, met.xb where xb is the json
builder, contains the fields to be indexed.
IndexResponse response = client.prepareIndex("nod",
"rel").setSource(met.xb).execute().actionGet();
I dont see any example in Client API regarding mapping.

How do I make met.xb.field("name", name) as the default ID?


On Thursday, 29 November 2012 15:08:02 UTC+5:30, Itamar Syn-Hershko
wrote:
> > > > You can use a hash of both as the document ID
  On Thu, Nov 29, 2012 at 11:12 AM, divyanshu das
divyan...@gmail.com wrote:
> > > > > I need to index data of 4 million users in
> > > > > elasticsearch using JAVA API.
    for each object these are the fields that are getting
indexed:-
    Name -
    Address -

    I need to put a check at the time of indexing. If another json
object with the same name as this object has already been indexed,
then either discard this object or overwrite the previous one.
    Can someone help me out here?




    --


  > > > >     > > > 
--
--
David Pilato
http://www.scrutmydocs.org/ http://www.scrutmydocs.org/
http://dev.david.pilato.fr/ http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

divyanshu_das · November 29, 2012, 11:39am

yes, I want to index with a given ID.

So, if the name is one single word "xyz", then it should work.

or if lets say I replace name with a particular ID (not Elasticsearch id
but my own id).

String newid = abc.getID();

IndexResponse response = client.prepareIndex("nod", "rel",
newid).setSource(met.xb).execute().actionGet();

Will this work?

On Thu, Nov 29, 2012 at 4:35 PM, David Pilato david@pilato.fr wrote:

**
If I understand what you are after (say index with a given ID): yes.
But, if name is a string like "Hello I'm happy to meet you!", It will not
work.
So you have to encode your ID.

To encode it, you can use something like :
https://github.com/scrutmydocs/scrutmydocs/blob/master/src/main/java/org/scrutmydocs/webapp/util/SignTool.java

Or perhaps : org.elasticsearch.common.UUID.fromString(name) will help;

David

Le 29 novembre 2012 à 11:49, divyanshu das divyanshu.das@gmail.com a
écrit :

@David,

so, lets say i define a new string,

String newid = abc.getName() // abc.getName() returns the name

IndexResponse response = client.prepareIndex("nod", "rel",
newid).setSource(met.xb).execute().actionGet();

Will this work?

On Thursday, 29 November 2012 15:46:14 UTC+5:30, David Pilato wrote:

Use something like

IndexResponse response = client.prepareIndex("nod",
"rel").setSource(met.xb).execute().actionGet();

where id is your name (or best as suggested by Itamar, a HASH of your
name).

HTH
David.

Le 29 novembre 2012 à 10:53, divyanshu das < divyan...@gmail.com> a
écrit :

I am new to this mapping concept on how and where to define it.

I am creating the index like this, met.xb where xb is the json builder,
contains the fields to be indexed.

IndexResponse response = client.prepareIndex("nod",
"rel").setSource(met.xb).execute().actionGet();

I dont see any example in Client API regarding mapping.

How do I make met.xb.field("name", name) as the default ID?

On Thursday, 29 November 2012 15:08:02 UTC+5:30, Itamar Syn-Hershko wrote:

You can use a hash of both as the document ID

On Thu, Nov 29, 2012 at 11:12 AM, divyanshu das divyan...@gmail.comwrote:

I need to index data of 4 million users in elasticsearch using JAVA API.

for each object these are the fields that are getting indexed:-

Name -
Address -

I need to put a check at the time of indexing. If another json object
with the same name as this object has already been indexed, then either
discard this object or overwrite the previous one.

Can someone help me out here?

--

--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

--

dadoonet · November 29, 2012, 2:47pm

Yes...

I would say: "just test it!" It's so easy to start an ES node in Java.

Have a look at: GitHub - elasticsearchfr/hands-on: Hands On Lab

It will help you to start in Java with ES.

HTH
David.

Le 29 novembre 2012 à 12:39, divyanshu das divyanshu.das@gmail.com a écrit :

yes, I want to index with a given ID.

So, if the name is one single word "xyz", then it should work.

or if lets say I replace name with a particular ID (not Elasticsearch id but
my own id).

String newid = abc.getID();

IndexResponse response = client.prepareIndex("nod", "rel",
newid).setSource(met.xb). execute().actionGet();

Will this work?

On Thu, Nov 29, 2012 at 4:35 PM, David Pilato <david@pilato.fr
mailto:david@pilato.fr > wrote:
If I understand what you are after (say index with a given ID): yes.
But, if name is a string like "Hello I'm happy to meet you!", It will not
work.
So you have to encode your ID.

To encode it, you can use something like :
https://github.com/scrutmydocs/scrutmydocs/blob/master/src/main/java/org/scrutmydocs/webapp/util/SignTool.java
https://github.com/scrutmydocs/scrutmydocs/blob/master/src/main/java/org/scrutmydocs/webapp/util/SignTool.java

Or perhaps : org.elasticsearch.common.UUID.fromString(name) will help;

David

Le 29 novembre 2012 à 11:49, divyanshu das < divyanshu.das@gmail.com
mailto:divyanshu.das@gmail.com > a écrit :
> > > @David,
so, lets say i define a new string,

String newid = abc.getName() // abc.getName() returns the name

IndexResponse response = client.prepareIndex("nod", "rel",
newid).setSource(met.xb).execute().actionGet();
Will this work?

On Thursday, 29 November 2012 15:46:14 UTC+5:30, David Pilato wrote:
  > > > > 
  Use something like

      > > > > > 
  > > > > IndexResponse response = client.prepareIndex("nod",
  > > > > "rel").setSource(met.xb).execute().actionGet();
  where id is your name (or best as suggested by Itamar, a HASH of
your name).
  HTH
  David.

  Le 29 novembre 2012 à 10:53, divyanshu das < divyan...@gmail.com>
a écrit :
   > > > > > I am new to this mapping concept on how and where to
   > > > > > define it.
   I am creating the index like this, met.xb where xb is the json
builder, contains the fields to be indexed.
   IndexResponse response = client.prepareIndex("nod",
"rel").setSource(met.xb).execute().actionGet();
   I dont see any example in Client API regarding mapping.

   How do I make met.xb.field("name", name) as the default ID?


   On Thursday, 29 November 2012 15:08:02 UTC+5:30, Itamar
Syn-Hershko wrote:
> > > > > > You can use a hash of both as the
> > > > > > document ID
     On Thu, Nov 29, 2012 at 11:12 AM, divyanshu das
divyan...@gmail.com wrote:
> > > > > > > I need to index data of 4 million users in
> > > > > > > elasticsearch using JAVA API.
       for each object these are the fields that are getting
indexed:-
       Name -
       Address -

       I need to put a check at the time of indexing. If
another json object with the same name as this object has already
been indexed, then either discard this object or overwrite the
previous one.
       Can someone help me out here?




       --


     > > > > > >        > > > > > 
   --



  > > > > 
  --
  David Pilato
  http://www.scrutmydocs.org/ <http://www.scrutmydocs.org/>
  http://dev.david.pilato.fr/ <http://dev.david.pilato.fr/>
  Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs
> > > 
--
--
David Pilato
http://www.scrutmydocs.org/ http://www.scrutmydocs.org/
http://dev.david.pilato.fr/ http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--
--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

How to avoid duplication of data while indexing

Best regards, Radu

Best regards,
Radu