Configure an index using a low level rest Client

gringogordo · October 18, 2017, 12:26pm

I am (re)writing an index creation app in java and struggling a little bit.

The existing app uses the TransportClient and I want to move to the High Level Rest Client (using the low level Rest Client to set it up). I want to create a new index and to do this I am following the migration documentation.

Starting with the sample code along these lines

			Settings indexSettings = Settings.builder() 
			        .put("index.number_of_shards", 1)
			        .put("index.number_of_replicas", 0)
			       .put("path.data", "NewDevCluster")  //THIS SETTING FAILS!
		               .build();

			String payload = XContentFactory.jsonBuilder() 
			        .startObject()
			            .startObject("settings") 
			                .value(indexSettings)
			            .endObject()
			            .startObject("mappings")  
			                .startObject("doc")
			                    .startObject("properties")
			                        .startObject("time")
			                            .field("type", "date")
			                        .endObject()
			                    .endObject()
			                .endObject()
			            .endObject()
			        .endObject().string();

			
			HttpEntity entityCreate = new NStringEntity(payload, ContentType.APPLICATION_JSON); 
							//CREATE INDEX
			Response response = restClient.performRequest("PUT", "newIndex", emptyMap(), entityCreate);

So...

This does the basics that I want (although I haven't figured out how to point it to a new cluster yet - in the long term I'm not sure whether I want to do this anyway maybe a new index or an alias will suffice ...)

However I have a general settings file (along these lines)

{
"index" : {
"analysis" : {
"char_filter" : {
"gref_mapping" : {
"type" : "mapping",
"mappings" : ["/=>E"]
}
},
"analyzer" : {
"gref_char_filter" : {
"tokenizer" : "standard",
"char_filter" : ["gref_mapping"]
}
}
}
}
}

which are being loaded into the transport client like

>  String settingsSource=getSourceFromMappingFile("settings.json");

 CreateIndexResponse resp = client.admin().indices().create(new CreateIndexRequest(INDEX_NAME).settings(settingsSource)).actionGet();

and a set of large mapping files which are loaded like this

client.admin().
            indices().
            preparePutMapping(indexName).
            setSource(getSourceFromMappingFile(mappingFile)).
            setType(indexType).
            execute().
            actionGet();

I'm struggling to move towards loading these using the restclients. Tbh I haven't tried very hard with the mapping files yet, and I can see how I could load them in a very long winded way but presumably there is a better way than creating an enourmous mapping section by loading 30 - 40 json documents into a XContentFactory.jsonBuilder ?) and I'm assuming there must be a way to load from file ?

Anyway what I'm asking is

Is it definitely possible to load the settings and mapping files ?
Can anyone point me in the right direction as to how I should do this

dadoonet · October 18, 2017, 1:06pm

Here is what I'm doing to create an index with settings (that I read previously from a file):

github.com

dadoonet/elasticsearch-beyonder/blob/master/src/main/java/fr/pilato/elasticsearch/tools/index/IndexElasticsearchUpdater.java#L278-L299


	private static void createIndexWithSettingsInElasticsearch(RestClient client, String index, String settings) throws Exception {
		logger.trace("createIndex([{}])", index);


		assert client != null;
		assert index != null;


		StringEntity entity = null;


		// If there are settings for this index, we use it. If not, using Elasticsearch defaults.
		if (settings != null) {
			logger.trace("Found settings for index [{}]: [{}]", index, settings);
			entity = new StringEntity(settings, ContentType.APPLICATION_JSON);
		}


		Response response = client.performRequest("PUT", "/" + index, Collections.<String, String>emptyMap(), entity);
		if (response.getStatusLine().getStatusCode() != 200) {
			logger.warn("Could not create index [{}]", index);
			throw new Exception("Could not create index ["+index+"].");
		}

This file has been truncated. show original

And same for add a mapping:

github.com

dadoonet/elasticsearch-beyonder/blob/master/src/main/java/fr/pilato/elasticsearch/tools/type/TypeElasticsearchUpdater.java#L228-L249


private static void createTypeWithMappingInElasticsearch(RestClient client, String index, String type, String mapping)
        throws Exception {
    logger.trace("createType([{}/{}])", index, type);


    assert client != null;
    assert index != null;
    assert type != null;


    if (mapping != null) {
        // Create type and mapping
        StringEntity entity = new StringEntity(mapping, ContentType.APPLICATION_JSON);
        Response response = client.performRequest("PUT", "/" + index + "/_mapping/" + type, Collections.<String, String>emptyMap(), entity);
        if (response.getStatusLine().getStatusCode() != 200) {
            logger.warn("Could not create type [{}/{}]", index, type);
            throw new Exception("Could not create type ["+index+"/"+type+"].");
        }
    } else {
        logger.trace("no content given for mapping. Ignoring type [{}/{}] creation.", index, type);
    }

This file has been truncated. show original

Though I think the later is useless as it's probably better to provide the mapping in the index creation call as at some point you will have only one type per index.

gringogordo · October 18, 2017, 1:17pm

That's great thanks I've been struggling with that for a day or more!

This is really helpful so no worries if you are too busy to answer but I'm interested by " at some point you will have only one type per index". In my current design I have 6 types in the index, is this a flawed design ?

( I need to revisit why I also have 20 odd mappings which are just sub-sets of one of the Types -I'm sure this is an inherited flaw which has just never been looked at since).

dadoonet · October 18, 2017, 2:33pm

In my current design I have 6 types in the index, is this a flawed design ?

Yes. From 6.0 you won't be able to create multiple type indices.

You can read that blog: Indices, types, and parent / child: current status and upcoming changes in Elasticsearch | Elastic Blog

gringogordo · October 18, 2017, 2:37pm

Wow, that passed me by completely (tbh I only started looking at upgrading elasticSearch as an afterthought so I didn't do my research!). Many thanks, my side project has just increased in complexity by several magnitudes!

gringogordo · October 19, 2017, 12:26pm

For future reference if anyone finds it useful I bumped into this just now which adds a lot of flesh to the core documentation

http://elasticsearch-cheatsheet.jolicode.com/

system · November 16, 2017, 12:26pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.