Creating, Updating, Validating and Rebuilding Index using Java API

Mxims_2 · January 22, 2013, 8:25am

Hi -

I am using TransportClient to connect to 3 nodes on my cluster. All nodes
are identical to each other.

Using Java API -

How do I create a new Index with a given type?

Once I create the index, I need to make sure that, it is up to date by
keeping with the transactions [CRUD]. I am using the following snippet to
add/update the documents into the index.

       bulkRequest.add(client.prepareIndex("supplier", type, id)
               .setSource(jsonBuilder()
                           .startObject()
                               .field("name", name)
                               .field("revision", revision)
                               .field("description", description)
                               .field("current", current)
                               .field("kindof", kindof)
                           .endObject()
                         )
               );

Using the lines below to delete documents when required
- bulkRequest.add(client.prepareDelete("supplier", type, id));
Once in a day, I need to make sure that the index is in sync with my
data in the database. To do this verification, I am planning to get all the
records from the db, and check whether they are available in the index.
Code snippet below..
SearchRequestBuilder searchBuilder =
client.prepareSearch("supplier").setTypes(type);
QueryBuilder qb = QueryBuilders.matchAllQuery();
- SearchResponse scrollResp =
  searchBuilder.setSearchType(SearchType.SCAN)
  .setScroll(new TimeValue(100))
  .setQuery(qb)
  .setSize(50).execute().actionGet();
  while (true) {
  scrollResp =
  client.prepareSearchScroll(scrollResp.getScrollId())
  .setScroll(new TimeValue(100))
  .execute()
  .actionGet();
```
         for (SearchHit hit : scrollResp.getHits()) {
             list.add(hit.id());

             names.add(hit.sourceAsMap().get("name"));
         }

         //Break condition: No hits are returned
         if (scrollResp.hits().hits().length == 0) {
             break;
         }
     }
```
- If there is mismatch, I need to rebuild the index from scratch. To
  do that I am currently deleting the mapping "type" from the "supplier"
  index.
- DeleteMappingRequest req = new DeleteMappingRequest("supplier");
  req.type(type);
  client.admin().indices().deleteMapping(req).actionGet();
Now I need to rebuild the index.. with type=Type above. How do I do
that?

Am I doing it the right way? Would there be any issues with my design? Is
deleting the mapping and creating the index with the deleted mapping again
causes any trouble?

Thank you,
Mxims

--

dadoonet · January 22, 2013, 8:39am

I think it's perfectly fine IMHO.
That's the way I did it on a previous project.

About the way you are going to create index and mappings:
If you use Spring, check this: GitHub - dadoonet/spring-elasticsearch: Spring factories for elasticsearch
https://github.com/dadoonet/spring-elasticsearch
If not, I recommend to create first your index and then to create all mappings
you need.

Create index :
https://github.com/dadoonet/spring-elasticsearch/blob/master/src/main/java/fr/pilato/spring/elasticsearch/ElasticsearchAbstractClientFactoryBean.java#L703
https://github.com/dadoonet/spring-elasticsearch/blob/master/src/main/java/fr/pilato/spring/elasticsearch/ElasticsearchAbstractClientFactoryBean.java#L703
Create mapping:
https://github.com/dadoonet/spring-elasticsearch/blob/master/src/main/java/fr/pilato/spring/elasticsearch/ElasticsearchAbstractClientFactoryBean.java#L667
https://github.com/dadoonet/spring-elasticsearch/blob/master/src/main/java/fr/pilato/spring/elasticsearch/ElasticsearchAbstractClientFactoryBean.java#L667

Does it help?

Le 22 janvier 2013 à 09:25, Mxims renjith.ck@gmail.com a écrit :

Hi -

I am using TransportClient to connect to 3 nodes on my cluster. All nodes are
identical to each other.

Using Java API -
* How do I create a new Index with a given type?
* Once I create the index, I need to make sure that, it is up to date by
keeping with the transactions [CRUD]. I am using the following snippet to
add/update the documents into the index.
             bulkRequest.add(client.prepareIndex("supplier", type, id)
                     .setSource(jsonBuilder()
                                 .startObject()
                                     .field("name", name)
                                     .field("revision", revision)
                                     .field("description", description)
                                     .field("current", current)
                                     .field("kindof", kindof)
                                 .endObject()
                               )
                     );

 * Using the lines below to delete documents when required
       o bulkRequest.add(client.prepareDelete("supplier", type, id));
 * Once in a day, I need to make sure that the index is in sync with my
data in the database. To do this verification, I am planning to get all the
records from the db, and check whether they are available in the index. Code
snippet below..
o SearchRequestBuilder searchBuilder =
client.prepareSearch("supplier").setTypes(type);
QueryBuilder qb = QueryBuilders.matchAllQuery();
o SearchResponse scrollResp =
searchBuilder.setSearchType(SearchType.SCAN)
.setScroll(new TimeValue(100))
.setQuery(qb)
.setSize(50).execute().actionGet();
while (true) {
scrollResp =
client.prepareSearchScroll(scrollResp.getScrollId())
.setScroll(new TimeValue(100))
.execute()
.actionGet();
                         for (SearchHit hit : scrollResp.getHits()) {
                             list.add(hit.id());

                             names.add(hit.sourceAsMap().get("name"));
                         }

                         //Break condition: No hits are returned
                         if (scrollResp.hits().hits().length == 0) {
                             break;
                         }
                     }
 * If there is mismatch, I need to rebuild the index from scratch. To do
that I am currently deleting the mapping "type" from the "supplier" index.
o DeleteMappingRequest req = new DeleteMappingRequest("supplier");
req.type(type);
client.admin().indices().deleteMapping(req).actionGet();
* Now I need to rebuild the index.. with type=Type above. How do I do
that?
Am I doing it the right way? Would there be any issues with my design? Is
deleting the mapping and creating the index with the deleted mapping again
causes any trouble?

Thank you,
Mxims

--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

Mxims_2 · January 23, 2013, 10:43pm

Hi David,

Thank you for your help.

I have been trying it from past two days.

Here is my scenario and what is going on with my index..

I have my index created with about 6 fields, all fields are strings. I am
using the TYPE as a key to identify different buckets of records. Type can
be 1234, 7890, 4567, etc. In my design, TYPE determines the bucket for each
records that are indexed. Users are restricted to search only within their
assigned bucket(s). i.e., While searching if I specify a given type (for
example, type=1234 or type=7890, etc.), result set should only contain
records that belong to that type.

When I am required to re-index the documents which belong to the type=1234,
at first I am dropping that mapping for type=1234 from the index, so that
it clears all the records that belong to that type. When I re-index the
records, for some reason records are updated in the index without having a
type. So I get a type missing exception. I am not sure whether I am
supposed to use the TYPE in this fashion to flag different buckets of
records.

Instead of deleting the type mapping, if I delete records one-by-one from
the index, and re-index I don't find any issues.

Any advice?

Thanks
Mxims

On Tuesday, January 22, 2013 12:39:30 AM UTC-8, David Pilato wrote:

I think it's perfectly fine IMHO.
That's the way I did it on a previous project.

About the way you are going to create index and mappings:
If you use Spring, check this:
GitHub - dadoonet/spring-elasticsearch: Spring factories for elasticsearch
If not, I recommend to create first your index and then to create all
mappings you need.

Create index :
https://github.com/dadoonet/spring-elasticsearch/blob/master/src/main/java/fr/pilato/spring/elasticsearch/ElasticsearchAbstractClientFactoryBean.java#L703
Create mapping:
https://github.com/dadoonet/spring-elasticsearch/blob/master/src/main/java/fr/pilato/spring/elasticsearch/ElasticsearchAbstractClientFactoryBean.java#L667

Does it help?

Le 22 janvier 2013 à 09:25, Mxims <renji...@gmail.com <javascript:>> a
écrit :

Hi -

I am using TransportClient to connect to 3 nodes on my cluster. All nodes
are identical to each other.

Using Java API -
How do I create a new Index with a given type?
Once I create the index, I need to make sure that, it is up to date
by keeping with the transactions [CRUD]. I am using the following snippet
to add/update the documents into the index.
       bulkRequest.add(client.prepareIndex("supplier", type, id)
               .setSource(jsonBuilder()
                           .startObject()
                               .field("name", name)
                               .field("revision", revision)
                               .field("description", description)
                               .field("current", current)
                               .field("kindof", kindof)
                           .endObject()
                         )
               );
Using the lines below to delete documents when required

bulkRequest.add(client.prepareDelete("supplier", type, id));
Once in a day, I need to make sure that the index is in sync with my
data in the database. To do this verification, I am planning to get all the
records from the db, and check whether they are available in the index.
Code snippet below..
SearchRequestBuilder searchBuilder =
client.prepareSearch("supplier").setTypes(type);
QueryBuilder qb = QueryBuilders.matchAllQuery();
SearchResponse scrollResp =
searchBuilder.setSearchType(SearchType.SCAN)
.setScroll(new TimeValue(100))
.setQuery(qb)
.setSize(50).execute().actionGet();
while (true) {
scrollResp =
client.prepareSearchScroll(scrollResp.getScrollId())
.setScroll(new TimeValue(100))
.execute()
.actionGet();
         for (SearchHit hit : scrollResp.getHits()) {
             list.add(hit.id());

             names.add(hit.sourceAsMap().get("name"));
         }

         //Break condition: No hits are returned
         if (scrollResp.hits().hits().length == 0) {
             break;
         }
     } 
If there is mismatch, I need to rebuild the index from scratch. To
do that I am currently deleting the mapping "type" from the "supplier"
index.

DeleteMappingRequest req = new DeleteMappingRequest("supplier");
req.type(type);
client.admin().indices().deleteMapping(req).actionGet();

Now I need to rebuild the index.. with type=Type above. How do I do
that?
Am I doing it the right way? Would there be any issues with my design? Is
deleting the mapping and creating the index with the deleted mapping again
causes any trouble?

Thank you,
Mxims

--

--
David Pilato
http://www.scrutmydocs.org/
http://dev.david.pilato.fr/
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

--

Clinton_Gormley · January 24, 2013, 4:39pm

Hiya Mxims

I have my index created with about 6 fields, all fields are strings. I
am using the TYPE as a key to identify different buckets of records.
Type can be 1234, 7890, 4567, etc. In my design, TYPE determines the
bucket for each records that are indexed. Users are restricted to
search only within their assigned bucket(s). i.e., While searching if
I specify a given type (for example, type=1234 or type=7890, etc.),
result set should only contain records that belong to that type.

When I am required to re-index the documents which belong to the
type=1234, at first I am dropping that mapping for type=1234 from the
index, so that it clears all the records that belong to that type.
When I re-index the records, for some reason records are updated in
the index without having a type. So I get a type missing exception. I
am not sure whether I am supposed to use the TYPE in this fashion to
flag different buckets of records.

Instead of deleting the type mapping, if I delete records one-by-one
from the index, and re-index I don't find any issues.

I tried the process you described above, and it all seems to work
normally. Possibly we're doing different things.

Please can you gist a curl recreation of what you're actually doing, to
show the problem that you encounter.

thanks

Clint

--

Mxims_2 · January 29, 2013, 1:31am

Hi Clint,

It seems the issue was related to deleting the mapping.. after I deleted
the mapping, I was not creating the mapping again before I indexed the new
documents.

After I did the put mapping, things seem to be working fine!

Thanks a lot for your help!

Thanks

On Thursday, January 24, 2013 8:39:34 AM UTC-8, Clinton Gormley wrote:

Hiya Mxims

I have my index created with about 6 fields, all fields are strings. I
am using the TYPE as a key to identify different buckets of records.
Type can be 1234, 7890, 4567, etc. In my design, TYPE determines the
bucket for each records that are indexed. Users are restricted to
search only within their assigned bucket(s). i.e., While searching if
I specify a given type (for example, type=1234 or type=7890, etc.),
result set should only contain records that belong to that type.

When I am required to re-index the documents which belong to the
type=1234, at first I am dropping that mapping for type=1234 from the
index, so that it clears all the records that belong to that type.
When I re-index the records, for some reason records are updated in
the index without having a type. So I get a type missing exception. I
am not sure whether I am supposed to use the TYPE in this fashion to
flag different buckets of records.

Instead of deleting the type mapping, if I delete records one-by-one
from the index, and re-index I don't find any issues.

I tried the process you described above, and it all seems to work
normally. Possibly we're doing different things.

Please can you gist a curl recreation of what you're actually doing, to
show the problem that you encounter.

thanks

Clint

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Topic		Replies	Views
Bulk + Update API Elasticsearch	3	429	August 16, 2018
Creating a new index using TransportClient/NodeClient using Java API in ES 2.1 Elasticsearch	1	797	July 5, 2017
Create index java API Elasticsearch	2	259	July 6, 2017
Bulk update with java API Elasticsearch	3	2457	July 6, 2017
ElasticSearch (0.17.6) BulkIndexer: PDF / HTML Docs Elasticsearch	1	373	July 6, 2017

Creating, Updating, Validating and Rebuilding Index using Java API

Related topics