Proper YAML Configuration of ElasticSearch


(Ronak Patel) #1

Hi All,

I'm fairly new to ElasticSearch and am looking for the proper way to
configure it to create my index and my index settings instead of relying on
the application to do it using the ES API.

Is there any documentation or any snippits someone could point me at to:

  1. Configure the Index Data Mappings and Create the Index

final CreateIndexRequestBuilder builder = this.getCreateIndexRequestBuilder
();

// create a mapping for a geocoordinatefinal Map<String, Object> source = new HashMap<String, Object>();final Map<String, Object> properties = new HashMap<String, Object>();final Map<String, Object> mapping = new HashMap<String, Object>();source.put("properties", properties);mapping.put("type", "geo_point");mapping.put("lat_lon", Boolean.TRUE);for (final String field : geoCoordinateFields) {

properties.put(field, mapping);

}builder.addMapping(this.documentType, source);// build the responsefinal CreateIndexResponse createResponse = builder.execute().get();if (createResponse.acknowledged()) {

DefaultSearchServiceImpl.LOGGER.info("Successfully created the index: {}", this.index);

} else {

DefaultSearchServiceImpl.LOGGER.error("Failed to create the index: {}", this.index);

}return createResponse.acknowledged();

  1. Handle Configuring the Lucene Analyzer
    final IndicesAdminClient indexService = this.searchService.admin().indices
    ();

final CreateIndexRequestBuilder createRequest = indexService.prepareCreate(this.index);// set the default analyzersfinal ImmutableSettings.Builder indexerSettings = ImmutableSettings.settingsBuilder();indexerSettings.put("analysis.analyzer.events.type", "custom");indexerSettings.put("analysis.analyzer.events.tokenizer", "standard");indexerSettings.put("analysis.analyzer.events.filter", "snowball, standard, lowercase");createRequest.setSettings(indexerSettings);return createRequest;

  1. What is the difference between the index name & index id? What is each of those used for?

Thanks!!

Ronak Patel


(Shay Banon) #2

Just build a json representing the settings, a json representing the mapping for each type, and load it. You don't have to use ES APis for it. Take your pick with whatever json library you wish to use.

On Wednesday, February 29, 2012 at 6:47 PM, Ronak Patel wrote:

Hi All,

I'm fairly new to ElasticSearch and am looking for the proper way to configure it to create my index and my index settings instead of relying on the application to do it using the ES API.

Is there any documentation or any snippits someone could point me at to:

  1. Configure the Index Data Mappings and Create the Index

final CreateIndexRequestBuilder builder = this.getCreateIndexRequestBuilder();
// create a mapping for a geocoordinate final Map<String, Object> source = new HashMap<String, Object>(); final Map<String, Object> properties = new HashMap<String, Object>(); final Map<String, Object> mapping = new HashMap<String, Object>(); source.put("properties", properties); mapping.put("type", "geo_point"); mapping.put("lat_lon", Boolean.TRUE); for (final String field : geoCoordinateFields) {

properties.put(field, mapping);

} builder.addMapping(this.documentType, source); // build the response final CreateIndexResponse createResponse = builder.execute().get(); if (createResponse.acknowledged()) {

DefaultSearchServiceImpl.LOGGER.info("Successfully created the index: {}", this.index);

} else {

DefaultSearchServiceImpl.LOGGER.error("Failed to create the index: {}", this.index);

} return createResponse.acknowledged();

  1. Handle Configuring the Lucene Analyzer
    final IndicesAdminClient indexService = this.searchService.admin().indices();
    final CreateIndexRequestBuilder createRequest = indexService.prepareCreate(this.index); // set the default analyzers final ImmutableSettings.Builder indexerSettings = ImmutableSettings.settingsBuilder(); indexerSettings.put("analysis.analyzer.events.type", "custom"); indexerSettings.put("analysis.analyzer.events.tokenizer", "standard"); indexerSettings.put("analysis.analyzer.events.filter", "snowball, standard, lowercase"); createRequest.setSettings(indexerSettings); return createRequest;

  2. What is the difference between the index name & index id? What is each of those used for?

Thanks!!

Ronak Patel


(Ronak Patel) #3

Thanks for your reply Shay.

Is this JSON something I would just paste into the config file? My set of
document types/field types are pretty static so I was hoping to just
configure it inside of ES and remove this extra initialization code.

On Thu, Mar 1, 2012 at 7:28 AM, Shay Banon kimchy@gmail.com wrote:

Just build a json representing the settings, a json representing the
mapping for each type, and load it. You don't have to use ES APis for it.
Take your pick with whatever json library you wish to use.

On Wednesday, February 29, 2012 at 6:47 PM, Ronak Patel wrote:

Hi All,

I'm fairly new to ElasticSearch and am looking for the proper way to
configure it to create my index and my index settings instead of relying on
the application to do it using the ES API.

Is there any documentation or any snippits someone could point me at to:

  1. Configure the Index Data Mappings and Create the Index

final CreateIndexRequestBuilder builder = this.
getCreateIndexRequestBuilder();

// create a mapping for a geocoordinatefinal Map<String, Object> source = new HashMap<String, Object>();final Map<String, Object> properties = new HashMap<String, Object>();final Map<String, Object> mapping = new HashMap<String, Object>();source.put("properties", properties);mapping.put("type", "geo_point");mapping.put("lat_lon", Boolean.TRUE);for (final String field : geoCoordinateFields) {

properties.put(field, mapping);

}builder.addMapping(this.documentType, source);// build the responsefinal CreateIndexResponse createResponse = builder.execute().get();if (createResponse.acknowledged()) {

DefaultSearchServiceImpl.LOGGER.info("Successfully created the index: {}", this.index);

} else {

DefaultSearchServiceImpl.LOGGER.error("Failed to create the index: {}", this.index);

}return createResponse.acknowledged();

  1. Handle Configuring the Lucene Analyzer
    final IndicesAdminClient indexService = this.searchService.admin().indices
    ();

final CreateIndexRequestBuilder createRequest = indexService.prepareCreate(this.index);// set the default analyzersfinal ImmutableSettings.Builder indexerSettings = ImmutableSettings.settingsBuilder();indexerSettings.put("analysis.analyzer.events.type", "custom");indexerSettings.put("analysis.analyzer.events.tokenizer", "standard");indexerSettings.put("analysis.analyzer.events.filter", "snowball, standard, lowercase");createRequest.setSettings(indexerSettings);return createRequest;

  1. What is the difference between the index name & index id? What is each of those used for?

Thanks!!

Ronak Patel


(Shay Banon) #4

You can load the relevant json form the classpath into a string, and then provide it as a string to the Java API.

On Tuesday, March 13, 2012 at 10:25 PM, Ronak Patel wrote:

Thanks for your reply Shay.

Is this JSON something I would just paste into the config file? My set of document types/field types are pretty static so I was hoping to just configure it inside of ES and remove this extra initialization code.

On Thu, Mar 1, 2012 at 7:28 AM, Shay Banon <kimchy@gmail.com (mailto:kimchy@gmail.com)> wrote:

Just build a json representing the settings, a json representing the mapping for each type, and load it. You don't have to use ES APis for it. Take your pick with whatever json library you wish to use.

On Wednesday, February 29, 2012 at 6:47 PM, Ronak Patel wrote:

Hi All,

I'm fairly new to ElasticSearch and am looking for the proper way to configure it to create my index and my index settings instead of relying on the application to do it using the ES API.

Is there any documentation or any snippits someone could point me at to:

  1. Configure the Index Data Mappings and Create the Index

final CreateIndexRequestBuilder builder = this.getCreateIndexRequestBuilder();
// create a mapping for a geocoordinate final Map<String, Object> source = new HashMap<String, Object>(); final Map<String, Object> properties = new HashMap<String, Object>(); final Map<String, Object> mapping = new HashMap<String, Object>(); source.put("properties", properties); mapping.put("type", "geo_point"); mapping.put("lat_lon", Boolean.TRUE); for (final String field : geoCoordinateFields) {

properties.put(field, mapping);

} builder.addMapping(this.documentType, source); // build the response final CreateIndexResponse createResponse = builder.execute().get(); if (createResponse.acknowledged()) {

DefaultSearchServiceImpl.LOGGER.info("Successfully created the index: {}", this.index);

} else {

DefaultSearchServiceImpl.LOGGER.error("Failed to create the index: {}", this.index);

} return createResponse.acknowledged();

  1. Handle Configuring the Lucene Analyzer
    final IndicesAdminClient indexService = this.searchService.admin().indices();
    final CreateIndexRequestBuilder createRequest = indexService.prepareCreate(this.index); // set the default analyzers final ImmutableSettings.Builder indexerSettings = ImmutableSettings.settingsBuilder(); indexerSettings.put("analysis.analyzer.events.type", "custom"); indexerSettings.put("analysis.analyzer.events.tokenizer", "standard"); indexerSettings.put("analysis.analyzer.events.filter", "snowball, standard, lowercase"); createRequest.setSettings(indexerSettings); return createRequest;

  2. What is the difference between the index name & index id? What is each of those used for?

Thanks!!

Ronak Patel


(Ronak Patel) #5

Right so that configuration is still in my client app instead of inside of
ES?? That's not what I was looking for...

On Wed, Mar 14, 2012 at 8:19 AM, Shay Banon kimchy@gmail.com wrote:

You can load the relevant json form the classpath into a string, and then
provide it as a string to the Java API.

On Tuesday, March 13, 2012 at 10:25 PM, Ronak Patel wrote:

Thanks for your reply Shay.

Is this JSON something I would just paste into the config file? My set of
document types/field types are pretty static so I was hoping to just
configure it inside of ES and remove this extra initialization code.

On Thu, Mar 1, 2012 at 7:28 AM, Shay Banon kimchy@gmail.com wrote:

Just build a json representing the settings, a json representing the
mapping for each type, and load it. You don't have to use ES APis for it.
Take your pick with whatever json library you wish to use.

On Wednesday, February 29, 2012 at 6:47 PM, Ronak Patel wrote:

Hi All,

I'm fairly new to ElasticSearch and am looking for the proper way to
configure it to create my index and my index settings instead of relying on
the application to do it using the ES API.

Is there any documentation or any snippits someone could point me at to:

  1. Configure the Index Data Mappings and Create the Index

final CreateIndexRequestBuilder builder = this.
getCreateIndexRequestBuilder();

// create a mapping for a geocoordinatefinal Map<String, Object> source = new HashMap<String, Object>();final Map<String, Object> properties = new HashMap<String, Object>();final Map<String, Object> mapping = new HashMap<String, Object>();source.put("properties", properties);mapping.put("type", "geo_point");mapping.put("lat_lon", Boolean.TRUE);for (final String field : geoCoordinateFields) {

properties.put(field, mapping);

}builder.addMapping(this.documentType, source);// build the responsefinal CreateIndexResponse createResponse = builder.execute().get();if (createResponse.acknowledged()) {

DefaultSearchServiceImpl.LOGGER.info("Successfully created the index: {}", this.index);

} else {

DefaultSearchServiceImpl.LOGGER.error("Failed to create the index: {}", this.index);

}return createResponse.acknowledged();

  1. Handle Configuring the Lucene Analyzer
    final IndicesAdminClient indexService = this.searchService.admin().indices
    ();

final CreateIndexRequestBuilder createRequest = indexService.prepareCreate(this.index);// set the default analyzersfinal ImmutableSettings.Builder indexerSettings = ImmutableSettings.settingsBuilder();indexerSettings.put("analysis.analyzer.events.type", "custom");indexerSettings.put("analysis.analyzer.events.tokenizer", "standard");indexerSettings.put("analysis.analyzer.events.filter", "snowball, standard, lowercase");createRequest.setSettings(indexerSettings);return createRequest;

  1. What is the difference between the index name & index id? What is each of those used for?

Thanks!!

Ronak Patel


(Shay Banon) #6

I don't really understand than what you after. The code you posted created
an index with its own specific settings from the "client" side. Can you
explicitly explain what you after?

On Thu, Mar 15, 2012 at 1:26 AM, Ronak Patel ronak2121@gmail.com wrote:

Right so that configuration is still in my client app instead of inside of
ES?? That's not what I was looking for...

On Wed, Mar 14, 2012 at 8:19 AM, Shay Banon kimchy@gmail.com wrote:

You can load the relevant json form the classpath into a string, and
then provide it as a string to the Java API.

On Tuesday, March 13, 2012 at 10:25 PM, Ronak Patel wrote:

Thanks for your reply Shay.

Is this JSON something I would just paste into the config file? My set of
document types/field types are pretty static so I was hoping to just
configure it inside of ES and remove this extra initialization code.

On Thu, Mar 1, 2012 at 7:28 AM, Shay Banon kimchy@gmail.com wrote:

Just build a json representing the settings, a json representing the
mapping for each type, and load it. You don't have to use ES APis for it.
Take your pick with whatever json library you wish to use.

On Wednesday, February 29, 2012 at 6:47 PM, Ronak Patel wrote:

Hi All,

I'm fairly new to ElasticSearch and am looking for the proper way to
configure it to create my index and my index settings instead of relying on
the application to do it using the ES API.

Is there any documentation or any snippits someone could point me at to:

  1. Configure the Index Data Mappings and Create the Index

final CreateIndexRequestBuilder builder = this.
getCreateIndexRequestBuilder();

// create a mapping for a geocoordinatefinal Map<String, Object> source = new HashMap<String, Object>();final Map<String, Object> properties = new HashMap<String, Object>();final Map<String, Object> mapping = new HashMap<String, Object>();source.put("properties", properties);mapping.put("type", "geo_point");mapping.put("lat_lon", Boolean.TRUE);for (final String field : geoCoordinateFields) {

properties.put(field, mapping);

}builder.addMapping(this.documentType, source);// build the responsefinal CreateIndexResponse createResponse = builder.execute().get();if (createResponse.acknowledged()) {

DefaultSearchServiceImpl.LOGGER.info("Successfully created the index: {}", this.index);

} else {

DefaultSearchServiceImpl.LOGGER.error("Failed to create the index: {}", this.index);

}return createResponse.acknowledged();

  1. Handle Configuring the Lucene Analyzer
    final IndicesAdminClient indexService = this.searchService.admin().
    indices();

final CreateIndexRequestBuilder createRequest = indexService.prepareCreate(this.index);// set the default analyzersfinal ImmutableSettings.Builder indexerSettings = ImmutableSettings.settingsBuilder();indexerSettings.put("analysis.analyzer.events.type", "custom");indexerSettings.put("analysis.analyzer.events.tokenizer", "standard");indexerSettings.put("analysis.analyzer.events.filter", "snowball, standard, lowercase");createRequest.setSettings(indexerSettings);return createRequest;

  1. What is the difference between the index name & index id? What is each of those used for?

Thanks!!

Ronak Patel


(Ronak Patel) #7

I want to get rid of this code from the "client" side and move it into the
Elastic Search configuration. I saw that the YAML config file has a way to
specify this for ES but I couldn't find any good examples on how to set it
up.

On Fri, Mar 16, 2012 at 1:05 PM, Shay Banon kimchy@gmail.com wrote:

I don't really understand than what you after. The code you posted created
an index with its own specific settings from the "client" side. Can you
explicitly explain what you after?

On Thu, Mar 15, 2012 at 1:26 AM, Ronak Patel ronak2121@gmail.com wrote:

Right so that configuration is still in my client app instead of inside
of ES?? That's not what I was looking for...

On Wed, Mar 14, 2012 at 8:19 AM, Shay Banon kimchy@gmail.com wrote:

You can load the relevant json form the classpath into a string, and
then provide it as a string to the Java API.

On Tuesday, March 13, 2012 at 10:25 PM, Ronak Patel wrote:

Thanks for your reply Shay.

Is this JSON something I would just paste into the config file? My set
of document types/field types are pretty static so I was hoping to just
configure it inside of ES and remove this extra initialization code.

On Thu, Mar 1, 2012 at 7:28 AM, Shay Banon kimchy@gmail.com wrote:

Just build a json representing the settings, a json representing the
mapping for each type, and load it. You don't have to use ES APis for it.
Take your pick with whatever json library you wish to use.

On Wednesday, February 29, 2012 at 6:47 PM, Ronak Patel wrote:

Hi All,

I'm fairly new to ElasticSearch and am looking for the proper way to
configure it to create my index and my index settings instead of relying on
the application to do it using the ES API.

Is there any documentation or any snippits someone could point me at to:

  1. Configure the Index Data Mappings and Create the Index

final CreateIndexRequestBuilder builder = this.
getCreateIndexRequestBuilder();

// create a mapping for a geocoordinatefinal Map<String, Object> source = new HashMap<String, Object>();final Map<String, Object> properties = new HashMap<String, Object>();final Map<String, Object> mapping = new HashMap<String, Object>();source.put("properties", properties);mapping.put("type", "geo_point");mapping.put("lat_lon", Boolean.TRUE);for (final String field : geoCoordinateFields) {

properties.put(field, mapping);

}builder.addMapping(this.documentType, source);// build the responsefinal CreateIndexResponse createResponse = builder.execute().get();if (createResponse.acknowledged()) {

DefaultSearchServiceImpl.LOGGER.info("Successfully created the index: {}", this.index);

} else {

DefaultSearchServiceImpl.LOGGER.error("Failed to create the index: {}", this.index);

}return createResponse.acknowledged();

  1. Handle Configuring the Lucene Analyzer
    final IndicesAdminClient indexService = this.searchService.admin().
    indices();

final CreateIndexRequestBuilder createRequest = indexService.prepareCreate(this.index);// set the default analyzersfinal ImmutableSettings.Builder indexerSettings = ImmutableSettings.settingsBuilder();indexerSettings.put("analysis.analyzer.events.type", "custom");indexerSettings.put("analysis.analyzer.events.tokenizer", "standard");indexerSettings.put("analysis.analyzer.events.filter", "snowball, standard, lowercase");createRequest.setSettings(indexerSettings);return createRequest;

  1. What is the difference between the index name & index id? What is each of those used for?

Thanks!!

Ronak Patel


(Shay Banon) #8

Yes, you can configure it in the elasticsearch.yml file (note, it can also
be a json one), or, you can configure in 0.19 index templates as part of
the configuration location (check at the bottom under Config):
http://www.elasticsearch.org/guide/reference/api/admin-indices-templates.html
.

On Mon, Mar 19, 2012 at 10:17 PM, Ronak Patel ronak2121@gmail.com wrote:

I want to get rid of this code from the "client" side and move it into the
Elastic Search configuration. I saw that the YAML config file has a way to
specify this for ES but I couldn't find any good examples on how to set it
up.

On Fri, Mar 16, 2012 at 1:05 PM, Shay Banon kimchy@gmail.com wrote:

I don't really understand than what you after. The code you posted
created an index with its own specific settings from the "client" side. Can
you explicitly explain what you after?

On Thu, Mar 15, 2012 at 1:26 AM, Ronak Patel ronak2121@gmail.com wrote:

Right so that configuration is still in my client app instead of inside
of ES?? That's not what I was looking for...

On Wed, Mar 14, 2012 at 8:19 AM, Shay Banon kimchy@gmail.com wrote:

You can load the relevant json form the classpath into a string, and
then provide it as a string to the Java API.

On Tuesday, March 13, 2012 at 10:25 PM, Ronak Patel wrote:

Thanks for your reply Shay.

Is this JSON something I would just paste into the config file? My set
of document types/field types are pretty static so I was hoping to just
configure it inside of ES and remove this extra initialization code.

On Thu, Mar 1, 2012 at 7:28 AM, Shay Banon kimchy@gmail.com wrote:

Just build a json representing the settings, a json representing the
mapping for each type, and load it. You don't have to use ES APis for it.
Take your pick with whatever json library you wish to use.

On Wednesday, February 29, 2012 at 6:47 PM, Ronak Patel wrote:

Hi All,

I'm fairly new to ElasticSearch and am looking for the proper way to
configure it to create my index and my index settings instead of relying on
the application to do it using the ES API.

Is there any documentation or any snippits someone could point me at to:

  1. Configure the Index Data Mappings and Create the Index

final CreateIndexRequestBuilder builder = this.
getCreateIndexRequestBuilder();

// create a mapping for a geocoordinatefinal Map<String, Object> source = new HashMap<String, Object>();final Map<String, Object> properties = new HashMap<String, Object>();final Map<String, Object> mapping = new HashMap<String, Object>();source.put("properties", properties);mapping.put("type", "geo_point");mapping.put("lat_lon", Boolean.TRUE);for (final String field : geoCoordinateFields) {

properties.put(field, mapping);

}builder.addMapping(this.documentType, source);// build the responsefinal CreateIndexResponse createResponse = builder.execute().get();if (createResponse.acknowledged()) {

DefaultSearchServiceImpl.LOGGER.info("Successfully created the index: {}", this.index);

} else {

DefaultSearchServiceImpl.LOGGER.error("Failed to create the index: {}", this.index);

}return createResponse.acknowledged();

  1. Handle Configuring the Lucene Analyzer
    final IndicesAdminClient indexService = this.searchService.admin().
    indices();

final CreateIndexRequestBuilder createRequest = indexService.prepareCreate(this.index);// set the default analyzersfinal ImmutableSettings.Builder indexerSettings = ImmutableSettings.settingsBuilder();indexerSettings.put("analysis.analyzer.events.type", "custom");indexerSettings.put("analysis.analyzer.events.tokenizer", "standard");indexerSettings.put("analysis.analyzer.events.filter", "snowball, standard, lowercase");createRequest.setSettings(indexerSettings);return createRequest;

  1. What is the difference between the index name & index id? What is each of those used for?

Thanks!!

Ronak Patel


(Ronak Patel) #9

Hi Shay,

Would it be possible for you to provide a more complete example?
I'm looking to specify the index name, document type, the fields in that
document + the geospatial fields in YAML.

Can you please have a sample that shows how I can do all of those things?

Thanks,

Ronak

On Tue, Mar 20, 2012 at 6:48 AM, Shay Banon kimchy@gmail.com wrote:

Yes, you can configure it in the elasticsearch.yml file (note, it can also
be a json one), or, you can configure in 0.19 index templates as part of
the configuration location (check at the bottom under Config):
http://www.elasticsearch.org/guide/reference/api/admin-indices-templates.html
.

On Mon, Mar 19, 2012 at 10:17 PM, Ronak Patel ronak2121@gmail.com wrote:

I want to get rid of this code from the "client" side and move it into
the Elastic Search configuration. I saw that the YAML config file has a way
to specify this for ES but I couldn't find any good examples on how to set
it up.

On Fri, Mar 16, 2012 at 1:05 PM, Shay Banon kimchy@gmail.com wrote:

I don't really understand than what you after. The code you posted
created an index with its own specific settings from the "client" side. Can
you explicitly explain what you after?

On Thu, Mar 15, 2012 at 1:26 AM, Ronak Patel ronak2121@gmail.comwrote:

Right so that configuration is still in my client app instead of inside
of ES?? That's not what I was looking for...

On Wed, Mar 14, 2012 at 8:19 AM, Shay Banon kimchy@gmail.com wrote:

You can load the relevant json form the classpath into a string, and
then provide it as a string to the Java API.

On Tuesday, March 13, 2012 at 10:25 PM, Ronak Patel wrote:

Thanks for your reply Shay.

Is this JSON something I would just paste into the config file? My set
of document types/field types are pretty static so I was hoping to just
configure it inside of ES and remove this extra initialization code.

On Thu, Mar 1, 2012 at 7:28 AM, Shay Banon kimchy@gmail.com wrote:

Just build a json representing the settings, a json representing the
mapping for each type, and load it. You don't have to use ES APis for it.
Take your pick with whatever json library you wish to use.

On Wednesday, February 29, 2012 at 6:47 PM, Ronak Patel wrote:

Hi All,

I'm fairly new to ElasticSearch and am looking for the proper way to
configure it to create my index and my index settings instead of relying on
the application to do it using the ES API.

Is there any documentation or any snippits someone could point me at
to:

  1. Configure the Index Data Mappings and Create the Index

final CreateIndexRequestBuilder builder = this.
getCreateIndexRequestBuilder();

// create a mapping for a geocoordinatefinal Map<String, Object> source = new HashMap<String, Object>();final Map<String, Object> properties = new HashMap<String, Object>();final Map<String, Object> mapping = new HashMap<String, Object>();source.put("properties", properties);mapping.put("type", "geo_point");mapping.put("lat_lon", Boolean.TRUE);for (final String field : geoCoordinateFields) {

properties.put(field, mapping);

}builder.addMapping(this.documentType, source);// build the responsefinal CreateIndexResponse createResponse = builder.execute().get();if (createResponse.acknowledged()) {

DefaultSearchServiceImpl.LOGGER.info("Successfully created the index: {}", this.index);

} else {

DefaultSearchServiceImpl.LOGGER.error("Failed to create the index: {}", this.index);

}return createResponse.acknowledged();

  1. Handle Configuring the Lucene Analyzer
    final IndicesAdminClient indexService = this.searchService.admin().
    indices();

final CreateIndexRequestBuilder createRequest = indexService.prepareCreate(this.index);// set the default analyzersfinal ImmutableSettings.Builder indexerSettings = ImmutableSettings.settingsBuilder();indexerSettings.put("analysis.analyzer.events.type", "custom");indexerSettings.put("analysis.analyzer.events.tokenizer", "standard");indexerSettings.put("analysis.analyzer.events.filter", "snowball, standard, lowercase");createRequest.setSettings(indexerSettings);return createRequest;

  1. What is the difference between the index name & index id? What is each of those used for?

Thanks!!

Ronak Patel


(Shay Banon) #10

If you want to provide an index level config, then you will need to do it
as part of an index template (with an example showed in my previous link).
The tempalte can simply match directly on the index name, and the index
level settings and mappings can eb set in the template.

On Sun, Apr 8, 2012 at 3:29 AM, Ronak Patel ronak2121@gmail.com wrote:

Hi Shay,

Would it be possible for you to provide a more complete example?
I'm looking to specify the index name, document type, the fields in that
document + the geospatial fields in YAML.

Can you please have a sample that shows how I can do all of those things?

Thanks,

Ronak

On Tue, Mar 20, 2012 at 6:48 AM, Shay Banon kimchy@gmail.com wrote:

Yes, you can configure it in the elasticsearch.yml file (note, it can
also be a json one), or, you can configure in 0.19 index templates as part
of the configuration location (check at the bottom under Config):
http://www.elasticsearch.org/guide/reference/api/admin-indices-templates.html
.

On Mon, Mar 19, 2012 at 10:17 PM, Ronak Patel ronak2121@gmail.comwrote:

I want to get rid of this code from the "client" side and move it into
the Elastic Search configuration. I saw that the YAML config file has a way
to specify this for ES but I couldn't find any good examples on how to set
it up.

On Fri, Mar 16, 2012 at 1:05 PM, Shay Banon kimchy@gmail.com wrote:

I don't really understand than what you after. The code you posted
created an index with its own specific settings from the "client" side. Can
you explicitly explain what you after?

On Thu, Mar 15, 2012 at 1:26 AM, Ronak Patel ronak2121@gmail.comwrote:

Right so that configuration is still in my client app instead of
inside of ES?? That's not what I was looking for...

On Wed, Mar 14, 2012 at 8:19 AM, Shay Banon kimchy@gmail.com wrote:

You can load the relevant json form the classpath into a string, and
then provide it as a string to the Java API.

On Tuesday, March 13, 2012 at 10:25 PM, Ronak Patel wrote:

Thanks for your reply Shay.

Is this JSON something I would just paste into the config file? My
set of document types/field types are pretty static so I was hoping to just
configure it inside of ES and remove this extra initialization code.

On Thu, Mar 1, 2012 at 7:28 AM, Shay Banon kimchy@gmail.com wrote:

Just build a json representing the settings, a json representing the
mapping for each type, and load it. You don't have to use ES APis for it.
Take your pick with whatever json library you wish to use.

On Wednesday, February 29, 2012 at 6:47 PM, Ronak Patel wrote:

Hi All,

I'm fairly new to ElasticSearch and am looking for the proper way to
configure it to create my index and my index settings instead of relying on
the application to do it using the ES API.

Is there any documentation or any snippits someone could point me at
to:

  1. Configure the Index Data Mappings and Create the Index

final CreateIndexRequestBuilder builder = this.
getCreateIndexRequestBuilder();

// create a mapping for a geocoordinatefinal Map<String, Object> source = new HashMap<String, Object>();final Map<String, Object> properties = new HashMap<String, Object>();final Map<String, Object> mapping = new HashMap<String, Object>();source.put("properties", properties);mapping.put("type", "geo_point");mapping.put("lat_lon", Boolean.TRUE);for (final String field : geoCoordinateFields) {

properties.put(field, mapping);

}builder.addMapping(this.documentType, source);// build the responsefinal CreateIndexResponse createResponse = builder.execute().get();if (createResponse.acknowledged()) {

DefaultSearchServiceImpl.LOGGER.info("Successfully created the index: {}", this.index);

} else {

DefaultSearchServiceImpl.LOGGER.error("Failed to create the index: {}", this.index);

}return createResponse.acknowledged();

  1. Handle Configuring the Lucene Analyzer
    final IndicesAdminClient indexService = this.searchService.admin().
    indices();

final CreateIndexRequestBuilder createRequest = indexService.prepareCreate(this.index);// set the default analyzersfinal ImmutableSettings.Builder indexerSettings = ImmutableSettings.settingsBuilder();indexerSettings.put("analysis.analyzer.events.type", "custom");indexerSettings.put("analysis.analyzer.events.tokenizer", "standard");indexerSettings.put("analysis.analyzer.events.filter", "snowball, standard, lowercase");createRequest.setSettings(indexerSettings);return createRequest;

  1. What is the difference between the index name & index id? What is each of those used for?

Thanks!!

Ronak Patel


(Ronak Patel) #11

So I set up a template and I see elasticsearch picking it up in the logs.
However, I'm getting errors when I run queries against this index about the
"startDate" or "coordinate" field.

Those errors are:

Caused by: org.elasticsearch.index.query.QueryParsingException: [events]
failed to find geo_point field [coordinate]
at
org.elasticsearch.index.query.GeoDistanceFilterParser.parse(GeoDistanceFilterParser.java:180)
at
org.elasticsearch.index.query.QueryParseContext.parseInnerFilter(QueryParseContext.java:224)
at
org.elasticsearch.index.query.IndexQueryParserService.parseInnerFilter(IndexQueryParserService.java:232)
at
org.elasticsearch.search.query.FilterParseElement.parse(FilterParseElement.java:33)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:545)

Caused by: org.elasticsearch.search.SearchParseException: [events][3]:
query[type:club_trip],from[0],size[10]: Parse Failure [No mapping found for
[startDate] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:164)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:138)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:70)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:545)

Here is my template file:

{
"template" : {
"template" : "events",
"settings" : {
"analysis.analyzer.events.type" : "custom",
"analysis.analyzer.events.tokenizer" : "standard",
"analysis.analyzer.events.filter" : "snowball,
standard, lowercase"
},
"mappings" : {
"events" : {
"source" : {
"eventId" : {
"type" : "int"
},
"coordinate" : {
"type" : "geo_point",
"lat_long" : true
},
"contact" : {
"type" : "string"
},
"costs" : {
"type" : "string"
},
"description" : {
"type" : "string"
},
"email" : {
"type" : "string"
},
"endDate" : {
"type" : "datetime"
},
"icon" : {
"type" : "string"
},
"location" : {
"type" : "string"
},
"name" : {
"type" : "string"
},
"phone" : {
"type" : "string"
},
"resortId" : {
"type" : "int"
},
"resortName" : {
"type" : "string"
},
"startDate" : {
"type" : "datetime"
},
"type" : {
"type" : "string"
},
"url" : {
"type" : "string"
}
}
}
}
}
}

This index does not contain any documents at the moment. But, I would
expect it to work fine as long as I had configured these mappings.

Is that not the case? (Or is there something wrong with my template file?)

Thanks,

Ronak Patel

On Sun, Apr 8, 2012 at 2:24 PM, Shay Banon kimchy@gmail.com wrote:

If you want to provide an index level config, then you will need to do it
as part of an index template (with an example showed in my previous link).
The tempalte can simply match directly on the index name, and the index
level settings and mappings can eb set in the template.

On Sun, Apr 8, 2012 at 3:29 AM, Ronak Patel ronak2121@gmail.com wrote:

Hi Shay,

Would it be possible for you to provide a more complete example?
I'm looking to specify the index name, document type, the fields in that
document + the geospatial fields in YAML.

Can you please have a sample that shows how I can do all of those things?

Thanks,

Ronak

On Tue, Mar 20, 2012 at 6:48 AM, Shay Banon kimchy@gmail.com wrote:

Yes, you can configure it in the elasticsearch.yml file (note, it can
also be a json one), or, you can configure in 0.19 index templates as part
of the configuration location (check at the bottom under Config):
http://www.elasticsearch.org/guide/reference/api/admin-indices-templates.html
.

On Mon, Mar 19, 2012 at 10:17 PM, Ronak Patel ronak2121@gmail.comwrote:

I want to get rid of this code from the "client" side and move it into
the Elastic Search configuration. I saw that the YAML config file has a way
to specify this for ES but I couldn't find any good examples on how to set
it up.

On Fri, Mar 16, 2012 at 1:05 PM, Shay Banon kimchy@gmail.com wrote:

I don't really understand than what you after. The code you posted
created an index with its own specific settings from the "client" side. Can
you explicitly explain what you after?

On Thu, Mar 15, 2012 at 1:26 AM, Ronak Patel ronak2121@gmail.comwrote:

Right so that configuration is still in my client app instead of
inside of ES?? That's not what I was looking for...

On Wed, Mar 14, 2012 at 8:19 AM, Shay Banon kimchy@gmail.com wrote:

You can load the relevant json form the classpath into a string,
and then provide it as a string to the Java API.

On Tuesday, March 13, 2012 at 10:25 PM, Ronak Patel wrote:

Thanks for your reply Shay.

Is this JSON something I would just paste into the config file? My
set of document types/field types are pretty static so I was hoping to just
configure it inside of ES and remove this extra initialization code.

On Thu, Mar 1, 2012 at 7:28 AM, Shay Banon kimchy@gmail.com wrote:

Just build a json representing the settings, a json representing
the mapping for each type, and load it. You don't have to use ES APis for
it. Take your pick with whatever json library you wish to use.

On Wednesday, February 29, 2012 at 6:47 PM, Ronak Patel wrote:

Hi All,

I'm fairly new to ElasticSearch and am looking for the proper way to
configure it to create my index and my index settings instead of relying on
the application to do it using the ES API.

Is there any documentation or any snippits someone could point me at
to:

  1. Configure the Index Data Mappings and Create the Index

final CreateIndexRequestBuilder builder = this.
getCreateIndexRequestBuilder();

// create a mapping for a geocoordinatefinal Map<String, Object> source = new HashMap<String, Object>();final Map<String, Object> properties = new HashMap<String, Object>();final Map<String, Object> mapping = new HashMap<String, Object>();source.put("properties", properties);mapping.put("type", "geo_point");mapping.put("lat_lon", Boolean.TRUE);for (final String field : geoCoordinateFields) {

properties.put(field, mapping);

}builder.addMapping(this.documentType, source);// build the responsefinal CreateIndexResponse createResponse = builder.execute().get();if (createResponse.acknowledged()) {

DefaultSearchServiceImpl.LOGGER.info("Successfully created the index: {}", this.index);

} else {

DefaultSearchServiceImpl.LOGGER.error("Failed to create the index: {}", this.index);

}return createResponse.acknowledged();

  1. Handle Configuring the Lucene Analyzer
    final IndicesAdminClient indexService = this.searchService.admin().
    indices();

final CreateIndexRequestBuilder createRequest = indexService.prepareCreate(this.index);// set the default analyzersfinal ImmutableSettings.Builder indexerSettings = ImmutableSettings.settingsBuilder();indexerSettings.put("analysis.analyzer.events.type", "custom");indexerSettings.put("analysis.analyzer.events.tokenizer", "standard");indexerSettings.put("analysis.analyzer.events.filter", "snowball, standard, lowercase");createRequest.setSettings(indexerSettings);return createRequest;

  1. What is the difference between the index name & index id? What is each of those used for?

Thanks!!

Ronak Patel


(Shay Banon) #12

It seems like your mappings are incorrect. Under an object level, there
should be "properties" element. So, if events is the type, and source is
the top level json object, both should have properties under them. Maybe
you can gist a sample json doc that shows the document you index, and then
helping with the mappings will be simpler. Also, you can use get mapping
API to see the automatic mapping created by elasticsearch for the data
indexed, and you can get a feeling for how it should look like.

On Mon, Apr 9, 2012 at 12:02 PM, Ronak Patel ronak2121@gmail.com wrote:

So I set up a template and I see elasticsearch picking it up in the logs.
However, I'm getting errors when I run queries against this index about the
"startDate" or "coordinate" field.

Those errors are:

Caused by: org.elasticsearch.index.query.QueryParsingException: [events]
failed to find geo_point field [coordinate]
at
org.elasticsearch.index.query.GeoDistanceFilterParser.parse(GeoDistanceFilterParser.java:180)
at
org.elasticsearch.index.query.QueryParseContext.parseInnerFilter(QueryParseContext.java:224)
at
org.elasticsearch.index.query.IndexQueryParserService.parseInnerFilter(IndexQueryParserService.java:232)
at
org.elasticsearch.search.query.FilterParseElement.parse(FilterParseElement.java:33)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:545)

Caused by: org.elasticsearch.search.SearchParseException: [events][3]:
query[type:club_trip],from[0],size[10]: Parse Failure [No mapping found for
[startDate] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:164)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:138)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:70)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:545)

Here is my template file:

{
"template" : {
"template" : "events",
"settings" : {
"analysis.analyzer.events.type" : "custom",
"analysis.analyzer.events.tokenizer" : "standard",
"analysis.analyzer.events.filter" : "snowball,
standard, lowercase"
},
"mappings" : {
"events" : {
"source" : {
"eventId" : {
"type" : "int"
},
"coordinate" : {
"type" : "geo_point",
"lat_long" : true
},
"contact" : {
"type" : "string"
},
"costs" : {
"type" : "string"
},
"description" : {
"type" : "string"
},
"email" : {
"type" : "string"
},
"endDate" : {
"type" : "datetime"
},
"icon" : {
"type" : "string"
},
"location" : {
"type" : "string"
},
"name" : {
"type" : "string"
},
"phone" : {
"type" : "string"
},
"resortId" : {
"type" : "int"
},
"resortName" : {
"type" : "string"
},
"startDate" : {
"type" : "datetime"
},
"type" : {
"type" : "string"
},
"url" : {
"type" : "string"
}
}
}
}
}
}

This index does not contain any documents at the moment. But, I would
expect it to work fine as long as I had configured these mappings.

Is that not the case? (Or is there something wrong with my template file?)

Thanks,

Ronak Patel

On Sun, Apr 8, 2012 at 2:24 PM, Shay Banon kimchy@gmail.com wrote:

If you want to provide an index level config, then you will need to do it
as part of an index template (with an example showed in my previous link).
The tempalte can simply match directly on the index name, and the index
level settings and mappings can eb set in the template.

On Sun, Apr 8, 2012 at 3:29 AM, Ronak Patel ronak2121@gmail.com wrote:

Hi Shay,

Would it be possible for you to provide a more complete example?
I'm looking to specify the index name, document type, the fields in that
document + the geospatial fields in YAML.

Can you please have a sample that shows how I can do all of those things?

Thanks,

Ronak

On Tue, Mar 20, 2012 at 6:48 AM, Shay Banon kimchy@gmail.com wrote:

Yes, you can configure it in the elasticsearch.yml file (note, it can
also be a json one), or, you can configure in 0.19 index templates as part
of the configuration location (check at the bottom under Config):
http://www.elasticsearch.org/guide/reference/api/admin-indices-templates.html
.

On Mon, Mar 19, 2012 at 10:17 PM, Ronak Patel ronak2121@gmail.comwrote:

I want to get rid of this code from the "client" side and move it into
the Elastic Search configuration. I saw that the YAML config file has a way
to specify this for ES but I couldn't find any good examples on how to set
it up.

On Fri, Mar 16, 2012 at 1:05 PM, Shay Banon kimchy@gmail.com wrote:

I don't really understand than what you after. The code you posted
created an index with its own specific settings from the "client" side. Can
you explicitly explain what you after?

On Thu, Mar 15, 2012 at 1:26 AM, Ronak Patel ronak2121@gmail.comwrote:

Right so that configuration is still in my client app instead of
inside of ES?? That's not what I was looking for...

On Wed, Mar 14, 2012 at 8:19 AM, Shay Banon kimchy@gmail.comwrote:

You can load the relevant json form the classpath into a string,
and then provide it as a string to the Java API.

On Tuesday, March 13, 2012 at 10:25 PM, Ronak Patel wrote:

Thanks for your reply Shay.

Is this JSON something I would just paste into the config file? My
set of document types/field types are pretty static so I was hoping to just
configure it inside of ES and remove this extra initialization code.

On Thu, Mar 1, 2012 at 7:28 AM, Shay Banon kimchy@gmail.comwrote:

Just build a json representing the settings, a json representing
the mapping for each type, and load it. You don't have to use ES APis for
it. Take your pick with whatever json library you wish to use.

On Wednesday, February 29, 2012 at 6:47 PM, Ronak Patel wrote:

Hi All,

I'm fairly new to ElasticSearch and am looking for the proper way
to configure it to create my index and my index settings instead of relying
on the application to do it using the ES API.

Is there any documentation or any snippits someone could point me
at to:

  1. Configure the Index Data Mappings and Create the Index

final CreateIndexRequestBuilder builder = this.
getCreateIndexRequestBuilder();

// create a mapping for a geocoordinatefinal Map<String, Object> source = new HashMap<String, Object>();final Map<String, Object> properties = new HashMap<String, Object>();final Map<String, Object> mapping = new HashMap<String, Object>();source.put("properties", properties);mapping.put("type", "geo_point");mapping.put("lat_lon", Boolean.TRUE);for (final String field : geoCoordinateFields) {

properties.put(field, mapping);

}builder.addMapping(this.documentType, source);// build the responsefinal CreateIndexResponse createResponse = builder.execute().get();if (createResponse.acknowledged()) {

DefaultSearchServiceImpl.LOGGER.info("Successfully created the index: {}", this.index);

} else {

DefaultSearchServiceImpl.LOGGER.error("Failed to create the index: {}", this.index);

}return createResponse.acknowledged();

  1. Handle Configuring the Lucene Analyzer
    final IndicesAdminClient indexService = this.searchService.admin().
    indices();

final CreateIndexRequestBuilder createRequest = indexService.prepareCreate(this.index);// set the default analyzersfinal ImmutableSettings.Builder indexerSettings = ImmutableSettings.settingsBuilder();indexerSettings.put("analysis.analyzer.events.type", "custom");indexerSettings.put("analysis.analyzer.events.tokenizer", "standard");indexerSettings.put("analysis.analyzer.events.filter", "snowball, standard, lowercase");createRequest.setSettings(indexerSettings);return createRequest;

  1. What is the difference between the index name & index id? What is each of those used for?

Thanks!!

Ronak Patel


(Ronak Patel) #13

Hi Shay,

Thanks a lot for helping me out.

I got this config file now:

{
"template": "events",
"settings": {
"analysis.analyzer.events.type": "custom",
"analysis.analyzer.events.tokenizer": "standard",
"analysis.analyzer.events.filter": "snowball, standard, lowercase"
},
"mappings": {
"events": {
"events": {
"properties": {
"startDate": {
"format": "dateOptionalTime",
"type": "date"
},
"icon": {
"type": "string"
},
"phone": {
"type": "string"
},
"coordinate": {
"lat_lon": true,
"type": "geo_point"
},
"location": {
"type": "string"
},
"endDate": {
"format": "dateOptionalTime",
"type": "date"
},
"type": {
"type": "string"
},
"contact": {
"type": "string"
},
"url": {
"type": "string"
},
"costs": {
"type": "string"
},
"eventId": {
"type": "long"
},
"email": {
"type": "string"
},
"description": {
"type": "string"
},
"resortName": {
"type": "string"
},
"name": {
"type": "string"
},
"resortId": {
"type": "long"
}
}
}
}
}
}

Yet I still get errors stating that mappings for startDate could not be
found. This is happening while I am trying to query against an index that's
been created but contains no documents.

On Wed, Apr 11, 2012 at 5:30 AM, Shay Banon kimchy@gmail.com wrote:

It seems like your mappings are incorrect. Under an object level, there
should be "properties" element. So, if events is the type, and source is
the top level json object, both should have properties under them. Maybe
you can gist a sample json doc that shows the document you index, and then
helping with the mappings will be simpler. Also, you can use get mapping
API to see the automatic mapping created by elasticsearch for the data
indexed, and you can get a feeling for how it should look like.

On Mon, Apr 9, 2012 at 12:02 PM, Ronak Patel ronak2121@gmail.com wrote:

So I set up a template and I see elasticsearch picking it up in the logs.
However, I'm getting errors when I run queries against this index about the
"startDate" or "coordinate" field.

Those errors are:

Caused by: org.elasticsearch.index.query.QueryParsingException: [events]
failed to find geo_point field [coordinate]
at
org.elasticsearch.index.query.GeoDistanceFilterParser.parse(GeoDistanceFilterParser.java:180)
at
org.elasticsearch.index.query.QueryParseContext.parseInnerFilter(QueryParseContext.java:224)
at
org.elasticsearch.index.query.IndexQueryParserService.parseInnerFilter(IndexQueryParserService.java:232)
at
org.elasticsearch.search.query.FilterParseElement.parse(FilterParseElement.java:33)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:545)

Caused by: org.elasticsearch.search.SearchParseException: [events][3]:
query[type:club_trip],from[0],size[10]: Parse Failure [No mapping found for
[startDate] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:164)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:138)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:70)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:545)

Here is my template file:

{
"template" : {
"template" : "events",
"settings" : {
"analysis.analyzer.events.type" : "custom",
"analysis.analyzer.events.tokenizer" : "standard",
"analysis.analyzer.events.filter" : "snowball,
standard, lowercase"
},
"mappings" : {
"events" : {
"source" : {
"eventId" : {
"type" : "int"
},
"coordinate" : {
"type" : "geo_point",
"lat_long" : true
},
"contact" : {
"type" : "string"
},
"costs" : {
"type" : "string"
},
"description" : {
"type" : "string"
},
"email" : {
"type" : "string"
},
"endDate" : {
"type" : "datetime"
},
"icon" : {
"type" : "string"
},
"location" : {
"type" : "string"
},
"name" : {
"type" : "string"
},
"phone" : {
"type" : "string"
},
"resortId" : {
"type" : "int"
},
"resortName" : {
"type" : "string"
},
"startDate" : {
"type" : "datetime"
},
"type" : {
"type" : "string"
},
"url" : {
"type" : "string"
}
}
}
}
}
}

This index does not contain any documents at the moment. But, I would
expect it to work fine as long as I had configured these mappings.

Is that not the case? (Or is there something wrong with my template file?)

Thanks,

Ronak Patel

On Sun, Apr 8, 2012 at 2:24 PM, Shay Banon kimchy@gmail.com wrote:

If you want to provide an index level config, then you will need to do
it as part of an index template (with an example showed in my previous
link). The tempalte can simply match directly on the index name, and the
index level settings and mappings can eb set in the template.

On Sun, Apr 8, 2012 at 3:29 AM, Ronak Patel ronak2121@gmail.com wrote:

Hi Shay,

Would it be possible for you to provide a more complete example?
I'm looking to specify the index name, document type, the fields in
that document + the geospatial fields in YAML.

Can you please have a sample that shows how I can do all of those
things?

Thanks,

Ronak

On Tue, Mar 20, 2012 at 6:48 AM, Shay Banon kimchy@gmail.com wrote:

Yes, you can configure it in the elasticsearch.yml file (note, it can
also be a json one), or, you can configure in 0.19 index templates as part
of the configuration location (check at the bottom under Config):
http://www.elasticsearch.org/guide/reference/api/admin-indices-templates.html
.

On Mon, Mar 19, 2012 at 10:17 PM, Ronak Patel ronak2121@gmail.comwrote:

I want to get rid of this code from the "client" side and move it
into the Elastic Search configuration. I saw that the YAML config file has
a way to specify this for ES but I couldn't find any good examples on how
to set it up.

On Fri, Mar 16, 2012 at 1:05 PM, Shay Banon kimchy@gmail.com wrote:

I don't really understand than what you after. The code you posted
created an index with its own specific settings from the "client" side. Can
you explicitly explain what you after?

On Thu, Mar 15, 2012 at 1:26 AM, Ronak Patel ronak2121@gmail.comwrote:

Right so that configuration is still in my client app instead of
inside of ES?? That's not what I was looking for...

On Wed, Mar 14, 2012 at 8:19 AM, Shay Banon kimchy@gmail.comwrote:

You can load the relevant json form the classpath into a string,
and then provide it as a string to the Java API.

On Tuesday, March 13, 2012 at 10:25 PM, Ronak Patel wrote:

Thanks for your reply Shay.

Is this JSON something I would just paste into the config file? My
set of document types/field types are pretty static so I was hoping to just
configure it inside of ES and remove this extra initialization code.

On Thu, Mar 1, 2012 at 7:28 AM, Shay Banon kimchy@gmail.comwrote:

Just build a json representing the settings, a json representing
the mapping for each type, and load it. You don't have to use ES APis for
it. Take your pick with whatever json library you wish to use.

On Wednesday, February 29, 2012 at 6:47 PM, Ronak Patel wrote:

Hi All,

I'm fairly new to ElasticSearch and am looking for the proper way
to configure it to create my index and my index settings instead of relying
on the application to do it using the ES API.

Is there any documentation or any snippits someone could point me
at to:

  1. Configure the Index Data Mappings and Create the Index

final CreateIndexRequestBuilder builder = this.
getCreateIndexRequestBuilder();

// create a mapping for a geocoordinatefinal Map<String, Object> source = new HashMap<String, Object>();final Map<String, Object> properties = new HashMap<String, Object>();final Map<String, Object> mapping = new HashMap<String, Object>();source.put("properties", properties);mapping.put("type", "geo_point");mapping.put("lat_lon", Boolean.TRUE);for (final String field : geoCoordinateFields) {

properties.put(field, mapping);

}builder.addMapping(this.documentType, source);// build the responsefinal CreateIndexResponse createResponse = builder.execute().get();if (createResponse.acknowledged()) {

DefaultSearchServiceImpl.LOGGER.info("Successfully created the index: {}", this.index);

} else {

DefaultSearchServiceImpl.LOGGER.error("Failed to create the index: {}", this.index);

}return createResponse.acknowledged();

  1. Handle Configuring the Lucene Analyzer
    final IndicesAdminClient indexService = this.searchService.admin
    ().indices();

final CreateIndexRequestBuilder createRequest = indexService.prepareCreate(this.index);// set the default analyzersfinal ImmutableSettings.Builder indexerSettings = ImmutableSettings.settingsBuilder();indexerSettings.put("analysis.analyzer.events.type", "custom");indexerSettings.put("analysis.analyzer.events.tokenizer", "standard");indexerSettings.put("analysis.analyzer.events.filter", "snowball, standard, lowercase");createRequest.setSettings(indexerSettings);return createRequest;

  1. What is the difference between the index name & index id? What is each of those used for?

Thanks!!

Ronak Patel


(Ronak Patel) #14

I do see in the logs that elasticsearch sets my settings so that part I
know works okay.
The mappings just don't seem to be getting set when I create a new index
called "events".

On Sat, Apr 14, 2012 at 12:40 AM, Ronak Patel ronak2121@gmail.com wrote:

Hi Shay,

Thanks a lot for helping me out.

I got this config file now:

{
"template": "events",
"settings": {
"analysis.analyzer.events.type": "custom",
"analysis.analyzer.events.tokenizer": "standard",
"analysis.analyzer.events.filter": "snowball, standard, lowercase"
},
"mappings": {
"events": {
"events": {
"properties": {
"startDate": {
"format": "dateOptionalTime",
"type": "date"
},
"icon": {
"type": "string"
},
"phone": {
"type": "string"
},
"coordinate": {
"lat_lon": true,
"type": "geo_point"
},
"location": {
"type": "string"
},
"endDate": {
"format": "dateOptionalTime",
"type": "date"
},
"type": {
"type": "string"
},
"contact": {
"type": "string"
},
"url": {
"type": "string"
},
"costs": {
"type": "string"
},
"eventId": {
"type": "long"
},
"email": {
"type": "string"
},
"description": {
"type": "string"
},
"resortName": {
"type": "string"
},
"name": {
"type": "string"
},
"resortId": {
"type": "long"
}
}
}
}
}
}

Yet I still get errors stating that mappings for startDate could not be
found. This is happening while I am trying to query against an index that's
been created but contains no documents.

On Wed, Apr 11, 2012 at 5:30 AM, Shay Banon kimchy@gmail.com wrote:

It seems like your mappings are incorrect. Under an object level, there
should be "properties" element. So, if events is the type, and source is
the top level json object, both should have properties under them. Maybe
you can gist a sample json doc that shows the document you index, and then
helping with the mappings will be simpler. Also, you can use get mapping
API to see the automatic mapping created by elasticsearch for the data
indexed, and you can get a feeling for how it should look like.

On Mon, Apr 9, 2012 at 12:02 PM, Ronak Patel ronak2121@gmail.com wrote:

So I set up a template and I see elasticsearch picking it up in the
logs. However, I'm getting errors when I run queries against this index
about the "startDate" or "coordinate" field.

Those errors are:

Caused by: org.elasticsearch.index.query.QueryParsingException: [events]
failed to find geo_point field [coordinate]
at
org.elasticsearch.index.query.GeoDistanceFilterParser.parse(GeoDistanceFilterParser.java:180)
at
org.elasticsearch.index.query.QueryParseContext.parseInnerFilter(QueryParseContext.java:224)
at
org.elasticsearch.index.query.IndexQueryParserService.parseInnerFilter(IndexQueryParserService.java:232)
at
org.elasticsearch.search.query.FilterParseElement.parse(FilterParseElement.java:33)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:545)

Caused by: org.elasticsearch.search.SearchParseException: [events][3]:
query[type:club_trip],from[0],size[10]: Parse Failure [No mapping found for
[startDate] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:164)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:138)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:70)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:545)

Here is my template file:

{
"template" : {
"template" : "events",
"settings" : {
"analysis.analyzer.events.type" : "custom",
"analysis.analyzer.events.tokenizer" :
"standard",
"analysis.analyzer.events.filter" : "snowball,
standard, lowercase"
},
"mappings" : {
"events" : {
"source" : {
"eventId" : {
"type" : "int"
},
"coordinate" : {
"type" : "geo_point",
"lat_long" : true
},
"contact" : {
"type" : "string"
},
"costs" : {
"type" : "string"
},
"description" : {
"type" : "string"
},
"email" : {
"type" : "string"
},
"endDate" : {
"type" : "datetime"
},
"icon" : {
"type" : "string"
},
"location" : {
"type" : "string"
},
"name" : {
"type" : "string"
},
"phone" : {
"type" : "string"
},
"resortId" : {
"type" : "int"
},
"resortName" : {
"type" : "string"
},
"startDate" : {
"type" : "datetime"
},
"type" : {
"type" : "string"
},
"url" : {
"type" : "string"
}
}
}
}
}
}

This index does not contain any documents at the moment. But, I would
expect it to work fine as long as I had configured these mappings.

Is that not the case? (Or is there something wrong with my template
file?)

Thanks,

Ronak Patel

On Sun, Apr 8, 2012 at 2:24 PM, Shay Banon kimchy@gmail.com wrote:

If you want to provide an index level config, then you will need to do
it as part of an index template (with an example showed in my previous
link). The tempalte can simply match directly on the index name, and the
index level settings and mappings can eb set in the template.

On Sun, Apr 8, 2012 at 3:29 AM, Ronak Patel ronak2121@gmail.comwrote:

Hi Shay,

Would it be possible for you to provide a more complete example?
I'm looking to specify the index name, document type, the fields in
that document + the geospatial fields in YAML.

Can you please have a sample that shows how I can do all of those
things?

Thanks,

Ronak

On Tue, Mar 20, 2012 at 6:48 AM, Shay Banon kimchy@gmail.com wrote:

Yes, you can configure it in the elasticsearch.yml file (note, it can
also be a json one), or, you can configure in 0.19 index templates as part
of the configuration location (check at the bottom under Config):
http://www.elasticsearch.org/guide/reference/api/admin-indices-templates.html
.

On Mon, Mar 19, 2012 at 10:17 PM, Ronak Patel ronak2121@gmail.comwrote:

I want to get rid of this code from the "client" side and move it
into the Elastic Search configuration. I saw that the YAML config file has
a way to specify this for ES but I couldn't find any good examples on how
to set it up.

On Fri, Mar 16, 2012 at 1:05 PM, Shay Banon kimchy@gmail.comwrote:

I don't really understand than what you after. The code you posted
created an index with its own specific settings from the "client" side. Can
you explicitly explain what you after?

On Thu, Mar 15, 2012 at 1:26 AM, Ronak Patel ronak2121@gmail.comwrote:

Right so that configuration is still in my client app instead of
inside of ES?? That's not what I was looking for...

On Wed, Mar 14, 2012 at 8:19 AM, Shay Banon kimchy@gmail.comwrote:

You can load the relevant json form the classpath into a string,
and then provide it as a string to the Java API.

On Tuesday, March 13, 2012 at 10:25 PM, Ronak Patel wrote:

Thanks for your reply Shay.

Is this JSON something I would just paste into the config file?
My set of document types/field types are pretty static so I was hoping to
just configure it inside of ES and remove this extra initialization code.

On Thu, Mar 1, 2012 at 7:28 AM, Shay Banon kimchy@gmail.comwrote:

Just build a json representing the settings, a json representing
the mapping for each type, and load it. You don't have to use ES APis for
it. Take your pick with whatever json library you wish to use.

On Wednesday, February 29, 2012 at 6:47 PM, Ronak Patel wrote:

Hi All,

I'm fairly new to ElasticSearch and am looking for the proper way
to configure it to create my index and my index settings instead of relying
on the application to do it using the ES API.

Is there any documentation or any snippits someone could point me
at to:

  1. Configure the Index Data Mappings and Create the Index

final CreateIndexRequestBuilder builder = this.
getCreateIndexRequestBuilder();

// create a mapping for a geocoordinatefinal Map<String, Object> source = new HashMap<String, Object>();final Map<String, Object> properties = new HashMap<String, Object>();final Map<String, Object> mapping = new HashMap<String, Object>();source.put("properties", properties);mapping.put("type", "geo_point");mapping.put("lat_lon", Boolean.TRUE);for (final String field : geoCoordinateFields) {

properties.put(field, mapping);

}builder.addMapping(this.documentType, source);// build the responsefinal CreateIndexResponse createResponse = builder.execute().get();if (createResponse.acknowledged()) {

DefaultSearchServiceImpl.LOGGER.info("Successfully created the index: {}", this.index);

} else {

DefaultSearchServiceImpl.LOGGER.error("Failed to create the index: {}", this.index);

}return createResponse.acknowledged();

  1. Handle Configuring the Lucene Analyzer
    final IndicesAdminClient indexService = this.searchService.admin
    ().indices();

final CreateIndexRequestBuilder createRequest = indexService.prepareCreate(this.index);// set the default analyzersfinal ImmutableSettings.Builder indexerSettings = ImmutableSettings.settingsBuilder();indexerSettings.put("analysis.analyzer.events.type", "custom");indexerSettings.put("analysis.analyzer.events.tokenizer", "standard");indexerSettings.put("analysis.analyzer.events.filter", "snowball, standard, lowercase");createRequest.setSettings(indexerSettings);return createRequest;

  1. What is the difference between the index name & index id? What is each of those used for?

Thanks!!

Ronak Patel


(Ronak Patel) #15

I put the admin config template in the /config/templates folder as stated
on the site. I assume that's the right spot?

On Sun, Apr 15, 2012 at 9:13 AM, Ronak Patel ronak2121@gmail.com wrote:

I do see in the logs that elasticsearch sets my settings so that part I
know works okay.
The mappings just don't seem to be getting set when I create a new index
called "events".

On Sat, Apr 14, 2012 at 12:40 AM, Ronak Patel ronak2121@gmail.com wrote:

Hi Shay,

Thanks a lot for helping me out.

I got this config file now:

{
"template": "events",
"settings": {
"analysis.analyzer.events.type": "custom",
"analysis.analyzer.events.tokenizer": "standard",
"analysis.analyzer.events.filter": "snowball, standard, lowercase"
},
"mappings": {
"events": {
"events": {
"properties": {
"startDate": {
"format": "dateOptionalTime",
"type": "date"
},
"icon": {
"type": "string"
},
"phone": {
"type": "string"
},
"coordinate": {
"lat_lon": true,
"type": "geo_point"
},
"location": {
"type": "string"
},
"endDate": {
"format": "dateOptionalTime",
"type": "date"
},
"type": {
"type": "string"
},
"contact": {
"type": "string"
},
"url": {
"type": "string"
},
"costs": {
"type": "string"
},
"eventId": {
"type": "long"
},
"email": {
"type": "string"
},
"description": {
"type": "string"
},
"resortName": {
"type": "string"
},
"name": {
"type": "string"
},
"resortId": {
"type": "long"
}
}
}
}
}
}

Yet I still get errors stating that mappings for startDate could not be
found. This is happening while I am trying to query against an index that's
been created but contains no documents.

On Wed, Apr 11, 2012 at 5:30 AM, Shay Banon kimchy@gmail.com wrote:

It seems like your mappings are incorrect. Under an object level, there
should be "properties" element. So, if events is the type, and source is
the top level json object, both should have properties under them. Maybe
you can gist a sample json doc that shows the document you index, and then
helping with the mappings will be simpler. Also, you can use get mapping
API to see the automatic mapping created by elasticsearch for the data
indexed, and you can get a feeling for how it should look like.

On Mon, Apr 9, 2012 at 12:02 PM, Ronak Patel ronak2121@gmail.comwrote:

So I set up a template and I see elasticsearch picking it up in the
logs. However, I'm getting errors when I run queries against this index
about the "startDate" or "coordinate" field.

Those errors are:

Caused by: org.elasticsearch.index.query.QueryParsingException:
[events] failed to find geo_point field [coordinate]
at
org.elasticsearch.index.query.GeoDistanceFilterParser.parse(GeoDistanceFilterParser.java:180)
at
org.elasticsearch.index.query.QueryParseContext.parseInnerFilter(QueryParseContext.java:224)
at
org.elasticsearch.index.query.IndexQueryParserService.parseInnerFilter(IndexQueryParserService.java:232)
at
org.elasticsearch.search.query.FilterParseElement.parse(FilterParseElement.java:33)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:545)

Caused by: org.elasticsearch.search.SearchParseException: [events][3]:
query[type:club_trip],from[0],size[10]: Parse Failure [No mapping found for
[startDate] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:164)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:138)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:70)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:545)

Here is my template file:

{
"template" : {
"template" : "events",
"settings" : {
"analysis.analyzer.events.type" : "custom",
"analysis.analyzer.events.tokenizer" :
"standard",
"analysis.analyzer.events.filter" : "snowball,
standard, lowercase"
},
"mappings" : {
"events" : {
"source" : {
"eventId" : {
"type" : "int"
},
"coordinate" : {
"type" : "geo_point",
"lat_long" : true
},
"contact" : {
"type" : "string"
},
"costs" : {
"type" : "string"
},
"description" : {
"type" : "string"
},
"email" : {
"type" : "string"
},
"endDate" : {
"type" : "datetime"
},
"icon" : {
"type" : "string"
},
"location" : {
"type" : "string"
},
"name" : {
"type" : "string"
},
"phone" : {
"type" : "string"
},
"resortId" : {
"type" : "int"
},
"resortName" : {
"type" : "string"
},
"startDate" : {
"type" : "datetime"
},
"type" : {
"type" : "string"
},
"url" : {
"type" : "string"
}
}
}
}
}
}

This index does not contain any documents at the moment. But, I would
expect it to work fine as long as I had configured these mappings.

Is that not the case? (Or is there something wrong with my template
file?)

Thanks,

Ronak Patel

On Sun, Apr 8, 2012 at 2:24 PM, Shay Banon kimchy@gmail.com wrote:

If you want to provide an index level config, then you will need to do
it as part of an index template (with an example showed in my previous
link). The tempalte can simply match directly on the index name, and the
index level settings and mappings can eb set in the template.

On Sun, Apr 8, 2012 at 3:29 AM, Ronak Patel ronak2121@gmail.comwrote:

Hi Shay,

Would it be possible for you to provide a more complete example?
I'm looking to specify the index name, document type, the fields in
that document + the geospatial fields in YAML.

Can you please have a sample that shows how I can do all of those
things?

Thanks,

Ronak

On Tue, Mar 20, 2012 at 6:48 AM, Shay Banon kimchy@gmail.com wrote:

Yes, you can configure it in the elasticsearch.yml file (note, it
can also be a json one), or, you can configure in 0.19 index templates as
part of the configuration location (check at the bottom under Config):
http://www.elasticsearch.org/guide/reference/api/admin-indices-templates.html
.

On Mon, Mar 19, 2012 at 10:17 PM, Ronak Patel ronak2121@gmail.comwrote:

I want to get rid of this code from the "client" side and move it
into the Elastic Search configuration. I saw that the YAML config file has
a way to specify this for ES but I couldn't find any good examples on how
to set it up.

On Fri, Mar 16, 2012 at 1:05 PM, Shay Banon kimchy@gmail.comwrote:

I don't really understand than what you after. The code you posted
created an index with its own specific settings from the "client" side. Can
you explicitly explain what you after?

On Thu, Mar 15, 2012 at 1:26 AM, Ronak Patel ronak2121@gmail.comwrote:

Right so that configuration is still in my client app instead of
inside of ES?? That's not what I was looking for...

On Wed, Mar 14, 2012 at 8:19 AM, Shay Banon kimchy@gmail.comwrote:

You can load the relevant json form the classpath into a
string, and then provide it as a string to the Java API.

On Tuesday, March 13, 2012 at 10:25 PM, Ronak Patel wrote:

Thanks for your reply Shay.

Is this JSON something I would just paste into the config file?
My set of document types/field types are pretty static so I was hoping to
just configure it inside of ES and remove this extra initialization code.

On Thu, Mar 1, 2012 at 7:28 AM, Shay Banon kimchy@gmail.comwrote:

Just build a json representing the settings, a json
representing the mapping for each type, and load it. You don't have to use
ES APis for it. Take your pick with whatever json library you wish to use.

On Wednesday, February 29, 2012 at 6:47 PM, Ronak Patel wrote:

Hi All,

I'm fairly new to ElasticSearch and am looking for the proper
way to configure it to create my index and my index settings instead of
relying on the application to do it using the ES API.

Is there any documentation or any snippits someone could point
me at to:

  1. Configure the Index Data Mappings and Create the Index

final CreateIndexRequestBuilder builder = this.
getCreateIndexRequestBuilder();

// create a mapping for a geocoordinatefinal Map<String, Object> source = new HashMap<String, Object>();final Map<String, Object> properties = new HashMap<String, Object>();final Map<String, Object> mapping = new HashMap<String, Object>();source.put("properties", properties);mapping.put("type", "geo_point");mapping.put("lat_lon", Boolean.TRUE);for (final String field : geoCoordinateFields) {

properties.put(field, mapping);

}builder.addMapping(this.documentType, source);// build the responsefinal CreateIndexResponse createResponse = builder.execute().get();if (createResponse.acknowledged()) {

DefaultSearchServiceImpl.LOGGER.info("Successfully created the index: {}", this.index);

} else {

DefaultSearchServiceImpl.LOGGER.error("Failed to create the index: {}", this.index);

}return createResponse.acknowledged();

  1. Handle Configuring the Lucene Analyzer
    final IndicesAdminClient indexService = this.searchService.admin
    ().indices();

final CreateIndexRequestBuilder createRequest = indexService.prepareCreate(this.index);// set the default analyzersfinal ImmutableSettings.Builder indexerSettings = ImmutableSettings.settingsBuilder();indexerSettings.put("analysis.analyzer.events.type", "custom");indexerSettings.put("analysis.analyzer.events.tokenizer", "standard");indexerSettings.put("analysis.analyzer.events.filter", "snowball, standard, lowercase");createRequest.setSettings(indexerSettings);return createRequest;

  1. What is the difference between the index name & index id? What is each of those used for?

Thanks!!

Ronak Patel


(Ronak Patel) #16

The logs also keep mentioning the word source. Is that some json element I
need to have in this mapping?

On Sun, Apr 15, 2012 at 9:15 AM, Ronak Patel ronak2121@gmail.com wrote:

I put the admin config template in the /config/templates folder as stated
on the site. I assume that's the right spot?

On Sun, Apr 15, 2012 at 9:13 AM, Ronak Patel ronak2121@gmail.com wrote:

I do see in the logs that elasticsearch sets my settings so that part I
know works okay.
The mappings just don't seem to be getting set when I create a new index
called "events".

On Sat, Apr 14, 2012 at 12:40 AM, Ronak Patel ronak2121@gmail.comwrote:

Hi Shay,

Thanks a lot for helping me out.

I got this config file now:

{
"template": "events",
"settings": {
"analysis.analyzer.events.type": "custom",
"analysis.analyzer.events.tokenizer": "standard",
"analysis.analyzer.events.filter": "snowball, standard,
lowercase"
},
"mappings": {
"events": {
"events": {
"properties": {
"startDate": {
"format": "dateOptionalTime",
"type": "date"
},
"icon": {
"type": "string"
},
"phone": {
"type": "string"
},
"coordinate": {
"lat_lon": true,
"type": "geo_point"
},
"location": {
"type": "string"
},
"endDate": {
"format": "dateOptionalTime",
"type": "date"
},
"type": {
"type": "string"
},
"contact": {
"type": "string"
},
"url": {
"type": "string"
},
"costs": {
"type": "string"
},
"eventId": {
"type": "long"
},
"email": {
"type": "string"
},
"description": {
"type": "string"
},
"resortName": {
"type": "string"
},
"name": {
"type": "string"
},
"resortId": {
"type": "long"
}
}
}
}
}
}

Yet I still get errors stating that mappings for startDate could not be
found. This is happening while I am trying to query against an index that's
been created but contains no documents.

On Wed, Apr 11, 2012 at 5:30 AM, Shay Banon kimchy@gmail.com wrote:

It seems like your mappings are incorrect. Under an object level, there
should be "properties" element. So, if events is the type, and source is
the top level json object, both should have properties under them. Maybe
you can gist a sample json doc that shows the document you index, and then
helping with the mappings will be simpler. Also, you can use get mapping
API to see the automatic mapping created by elasticsearch for the data
indexed, and you can get a feeling for how it should look like.

On Mon, Apr 9, 2012 at 12:02 PM, Ronak Patel ronak2121@gmail.comwrote:

So I set up a template and I see elasticsearch picking it up in the
logs. However, I'm getting errors when I run queries against this index
about the "startDate" or "coordinate" field.

Those errors are:

Caused by: org.elasticsearch.index.query.QueryParsingException:
[events] failed to find geo_point field [coordinate]
at
org.elasticsearch.index.query.GeoDistanceFilterParser.parse(GeoDistanceFilterParser.java:180)
at
org.elasticsearch.index.query.QueryParseContext.parseInnerFilter(QueryParseContext.java:224)
at
org.elasticsearch.index.query.IndexQueryParserService.parseInnerFilter(IndexQueryParserService.java:232)
at
org.elasticsearch.search.query.FilterParseElement.parse(FilterParseElement.java:33)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:545)

Caused by: org.elasticsearch.search.SearchParseException: [events][3]:
query[type:club_trip],from[0],size[10]: Parse Failure [No mapping found for
[startDate] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:164)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:138)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:70)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:545)

Here is my template file:

{
"template" : {
"template" : "events",
"settings" : {
"analysis.analyzer.events.type" : "custom",
"analysis.analyzer.events.tokenizer" :
"standard",
"analysis.analyzer.events.filter" : "snowball,
standard, lowercase"
},
"mappings" : {
"events" : {
"source" : {
"eventId" : {
"type" : "int"
},
"coordinate" : {
"type" : "geo_point",
"lat_long" : true
},
"contact" : {
"type" : "string"
},
"costs" : {
"type" : "string"
},
"description" : {
"type" : "string"
},
"email" : {
"type" : "string"
},
"endDate" : {
"type" : "datetime"
},
"icon" : {
"type" : "string"
},
"location" : {
"type" : "string"
},
"name" : {
"type" : "string"
},
"phone" : {
"type" : "string"
},
"resortId" : {
"type" : "int"
},
"resortName" : {
"type" : "string"
},
"startDate" : {
"type" : "datetime"
},
"type" : {
"type" : "string"
},
"url" : {
"type" : "string"
}
}
}
}
}
}

This index does not contain any documents at the moment. But, I would
expect it to work fine as long as I had configured these mappings.

Is that not the case? (Or is there something wrong with my template
file?)

Thanks,

Ronak Patel

On Sun, Apr 8, 2012 at 2:24 PM, Shay Banon kimchy@gmail.com wrote:

If you want to provide an index level config, then you will need to
do it as part of an index template (with an example showed in my previous
link). The tempalte can simply match directly on the index name, and the
index level settings and mappings can eb set in the template.

On Sun, Apr 8, 2012 at 3:29 AM, Ronak Patel ronak2121@gmail.comwrote:

Hi Shay,

Would it be possible for you to provide a more complete example?
I'm looking to specify the index name, document type, the fields in
that document + the geospatial fields in YAML.

Can you please have a sample that shows how I can do all of those
things?

Thanks,

Ronak

On Tue, Mar 20, 2012 at 6:48 AM, Shay Banon kimchy@gmail.comwrote:

Yes, you can configure it in the elasticsearch.yml file (note, it
can also be a json one), or, you can configure in 0.19 index templates as
part of the configuration location (check at the bottom under Config):
http://www.elasticsearch.org/guide/reference/api/admin-indices-templates.html
.

On Mon, Mar 19, 2012 at 10:17 PM, Ronak Patel ronak2121@gmail.comwrote:

I want to get rid of this code from the "client" side and move it
into the Elastic Search configuration. I saw that the YAML config file has
a way to specify this for ES but I couldn't find any good examples on how
to set it up.

On Fri, Mar 16, 2012 at 1:05 PM, Shay Banon kimchy@gmail.comwrote:

I don't really understand than what you after. The code you
posted created an index with its own specific settings from the "client"
side. Can you explicitly explain what you after?

On Thu, Mar 15, 2012 at 1:26 AM, Ronak Patel <ronak2121@gmail.com

wrote:

Right so that configuration is still in my client app instead of
inside of ES?? That's not what I was looking for...

On Wed, Mar 14, 2012 at 8:19 AM, Shay Banon kimchy@gmail.comwrote:

You can load the relevant json form the classpath into a
string, and then provide it as a string to the Java API.

On Tuesday, March 13, 2012 at 10:25 PM, Ronak Patel wrote:

Thanks for your reply Shay.

Is this JSON something I would just paste into the config file?
My set of document types/field types are pretty static so I was hoping to
just configure it inside of ES and remove this extra initialization code.

On Thu, Mar 1, 2012 at 7:28 AM, Shay Banon kimchy@gmail.comwrote:

Just build a json representing the settings, a json
representing the mapping for each type, and load it. You don't have to use
ES APis for it. Take your pick with whatever json library you wish to use.

On Wednesday, February 29, 2012 at 6:47 PM, Ronak Patel wrote:

Hi All,

I'm fairly new to ElasticSearch and am looking for the proper
way to configure it to create my index and my index settings instead of
relying on the application to do it using the ES API.

Is there any documentation or any snippits someone could point
me at to:

  1. Configure the Index Data Mappings and Create the Index

final CreateIndexRequestBuilder builder = this.
getCreateIndexRequestBuilder();

// create a mapping for a geocoordinatefinal Map<String, Object> source = new HashMap<String, Object>();final Map<String, Object> properties = new HashMap<String, Object>();final Map<String, Object> mapping = new HashMap<String, Object>();source.put("properties", properties);mapping.put("type", "geo_point");mapping.put("lat_lon", Boolean.TRUE);for (final String field : geoCoordinateFields) {

properties.put(field, mapping);

}builder.addMapping(this.documentType, source);// build the responsefinal CreateIndexResponse createResponse = builder.execute().get();if (createResponse.acknowledged()) {

DefaultSearchServiceImpl.LOGGER.info("Successfully created the index: {}", this.index);

} else {

DefaultSearchServiceImpl.LOGGER.error("Failed to create the index: {}", this.index);

}return createResponse.acknowledged();

  1. Handle Configuring the Lucene Analyzer
    final IndicesAdminClient indexService = this.searchService.
    admin().indices();

final CreateIndexRequestBuilder createRequest = indexService.prepareCreate(this.index);// set the default analyzersfinal ImmutableSettings.Builder indexerSettings = ImmutableSettings.settingsBuilder();indexerSettings.put("analysis.analyzer.events.type", "custom");indexerSettings.put("analysis.analyzer.events.tokenizer", "standard");indexerSettings.put("analysis.analyzer.events.filter", "snowball, standard, lowercase");createRequest.setSettings(indexerSettings);return createRequest;

  1. What is the difference between the index name & index id? What is each of those used for?

Thanks!!

Ronak Patel


(Ronak Patel) #17

I posted the mapping and settings when I created the index and it seems
that elasticsearch has accepted them.
So, that tells me that something is wrong with how elasticsearch is picking
up the template...

On Sun, Apr 15, 2012 at 12:28 PM, Ronak Patel ronak2121@gmail.com wrote:

The logs also keep mentioning the word source. Is that some json element I
need to have in this mapping?

On Sun, Apr 15, 2012 at 9:15 AM, Ronak Patel ronak2121@gmail.com wrote:

I put the admin config template in the /config/templates folder as stated
on the site. I assume that's the right spot?

On Sun, Apr 15, 2012 at 9:13 AM, Ronak Patel ronak2121@gmail.com wrote:

I do see in the logs that elasticsearch sets my settings so that part I
know works okay.
The mappings just don't seem to be getting set when I create a new index
called "events".

On Sat, Apr 14, 2012 at 12:40 AM, Ronak Patel ronak2121@gmail.comwrote:

Hi Shay,

Thanks a lot for helping me out.

I got this config file now:

{
"template": "events",
"settings": {
"analysis.analyzer.events.type": "custom",
"analysis.analyzer.events.tokenizer": "standard",
"analysis.analyzer.events.filter": "snowball, standard,
lowercase"
},
"mappings": {
"events": {
"events": {
"properties": {
"startDate": {
"format": "dateOptionalTime",
"type": "date"
},
"icon": {
"type": "string"
},
"phone": {
"type": "string"
},
"coordinate": {
"lat_lon": true,
"type": "geo_point"
},
"location": {
"type": "string"
},
"endDate": {
"format": "dateOptionalTime",
"type": "date"
},
"type": {
"type": "string"
},
"contact": {
"type": "string"
},
"url": {
"type": "string"
},
"costs": {
"type": "string"
},
"eventId": {
"type": "long"
},
"email": {
"type": "string"
},
"description": {
"type": "string"
},
"resortName": {
"type": "string"
},
"name": {
"type": "string"
},
"resortId": {
"type": "long"
}
}
}
}
}
}

Yet I still get errors stating that mappings for startDate could not be
found. This is happening while I am trying to query against an index that's
been created but contains no documents.

On Wed, Apr 11, 2012 at 5:30 AM, Shay Banon kimchy@gmail.com wrote:

It seems like your mappings are incorrect. Under an object level,
there should be "properties" element. So, if events is the type, and source
is the top level json object, both should have properties under them. Maybe
you can gist a sample json doc that shows the document you index, and then
helping with the mappings will be simpler. Also, you can use get mapping
API to see the automatic mapping created by elasticsearch for the data
indexed, and you can get a feeling for how it should look like.

On Mon, Apr 9, 2012 at 12:02 PM, Ronak Patel ronak2121@gmail.comwrote:

So I set up a template and I see elasticsearch picking it up in the
logs. However, I'm getting errors when I run queries against this index
about the "startDate" or "coordinate" field.

Those errors are:

Caused by: org.elasticsearch.index.query.QueryParsingException:
[events] failed to find geo_point field [coordinate]
at
org.elasticsearch.index.query.GeoDistanceFilterParser.parse(GeoDistanceFilterParser.java:180)
at
org.elasticsearch.index.query.QueryParseContext.parseInnerFilter(QueryParseContext.java:224)
at
org.elasticsearch.index.query.IndexQueryParserService.parseInnerFilter(IndexQueryParserService.java:232)
at
org.elasticsearch.search.query.FilterParseElement.parse(FilterParseElement.java:33)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:545)

Caused by: org.elasticsearch.search.SearchParseException:
[events][3]: query[type:club_trip],from[0],size[10]: Parse Failure [No
mapping found for [startDate] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:164)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:138)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:70)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:545)

Here is my template file:

{
"template" : {
"template" : "events",
"settings" : {
"analysis.analyzer.events.type" : "custom",
"analysis.analyzer.events.tokenizer" :
"standard",
"analysis.analyzer.events.filter" :
"snowball, standard, lowercase"
},
"mappings" : {
"events" : {
"source" : {
"eventId" : {
"type" : "int"
},
"coordinate" : {
"type" : "geo_point",
"lat_long" : true
},
"contact" : {
"type" : "string"
},
"costs" : {
"type" : "string"
},
"description" : {
"type" : "string"
},
"email" : {
"type" : "string"
},
"endDate" : {
"type" : "datetime"
},
"icon" : {
"type" : "string"
},
"location" : {
"type" : "string"
},
"name" : {
"type" : "string"
},
"phone" : {
"type" : "string"
},
"resortId" : {
"type" : "int"
},
"resortName" : {
"type" : "string"
},
"startDate" : {
"type" : "datetime"
},
"type" : {
"type" : "string"
},
"url" : {
"type" : "string"
}
}
}
}
}
}

This index does not contain any documents at the moment. But, I would
expect it to work fine as long as I had configured these mappings.

Is that not the case? (Or is there something wrong with my template
file?)

Thanks,

Ronak Patel

On Sun, Apr 8, 2012 at 2:24 PM, Shay Banon kimchy@gmail.com wrote:

If you want to provide an index level config, then you will need to
do it as part of an index template (with an example showed in my previous
link). The tempalte can simply match directly on the index name, and the
index level settings and mappings can eb set in the template.

On Sun, Apr 8, 2012 at 3:29 AM, Ronak Patel ronak2121@gmail.comwrote:

Hi Shay,

Would it be possible for you to provide a more complete example?
I'm looking to specify the index name, document type, the fields in
that document + the geospatial fields in YAML.

Can you please have a sample that shows how I can do all of those
things?

Thanks,

Ronak

On Tue, Mar 20, 2012 at 6:48 AM, Shay Banon kimchy@gmail.comwrote:

Yes, you can configure it in the elasticsearch.yml file (note, it
can also be a json one), or, you can configure in 0.19 index templates as
part of the configuration location (check at the bottom under Config):
http://www.elasticsearch.org/guide/reference/api/admin-indices-templates.html
.

On Mon, Mar 19, 2012 at 10:17 PM, Ronak Patel <ronak2121@gmail.com

wrote:

I want to get rid of this code from the "client" side and move it
into the Elastic Search configuration. I saw that the YAML config file has
a way to specify this for ES but I couldn't find any good examples on how
to set it up.

On Fri, Mar 16, 2012 at 1:05 PM, Shay Banon kimchy@gmail.comwrote:

I don't really understand than what you after. The code you
posted created an index with its own specific settings from the "client"
side. Can you explicitly explain what you after?

On Thu, Mar 15, 2012 at 1:26 AM, Ronak Patel <
ronak2121@gmail.com> wrote:

Right so that configuration is still in my client app instead
of inside of ES?? That's not what I was looking for...

On Wed, Mar 14, 2012 at 8:19 AM, Shay Banon kimchy@gmail.comwrote:

You can load the relevant json form the classpath into a
string, and then provide it as a string to the Java API.

On Tuesday, March 13, 2012 at 10:25 PM, Ronak Patel wrote:

Thanks for your reply Shay.

Is this JSON something I would just paste into the config
file? My set of document types/field types are pretty static so I was
hoping to just configure it inside of ES and remove this extra
initialization code.

On Thu, Mar 1, 2012 at 7:28 AM, Shay Banon kimchy@gmail.comwrote:

Just build a json representing the settings, a json
representing the mapping for each type, and load it. You don't have to use
ES APis for it. Take your pick with whatever json library you wish to use.

On Wednesday, February 29, 2012 at 6:47 PM, Ronak Patel wrote:

Hi All,

I'm fairly new to ElasticSearch and am looking for the proper
way to configure it to create my index and my index settings instead of
relying on the application to do it using the ES API.

Is there any documentation or any snippits someone could point
me at to:

  1. Configure the Index Data Mappings and Create the Index

final CreateIndexRequestBuilder builder = this.
getCreateIndexRequestBuilder();

// create a mapping for a geocoordinatefinal Map<String, Object> source = new HashMap<String, Object>();final Map<String, Object> properties = new HashMap<String, Object>();final Map<String, Object> mapping = new HashMap<String, Object>();source.put("properties", properties);mapping.put("type", "geo_point");mapping.put("lat_lon", Boolean.TRUE);for (final String field : geoCoordinateFields) {

properties.put(field, mapping);

}builder.addMapping(this.documentType, source);// build the responsefinal CreateIndexResponse createResponse = builder.execute().get();if (createResponse.acknowledged()) {

DefaultSearchServiceImpl.LOGGER.info("Successfully created the index: {}", this.index);

} else {

DefaultSearchServiceImpl.LOGGER.error("Failed to create the index: {}", this.index);

}return createResponse.acknowledged();

  1. Handle Configuring the Lucene Analyzer
    final IndicesAdminClient indexService = this.searchService.
    admin().indices();

final CreateIndexRequestBuilder createRequest = indexService.prepareCreate(this.index);// set the default analyzersfinal ImmutableSettings.Builder indexerSettings = ImmutableSettings.settingsBuilder();indexerSettings.put("analysis.analyzer.events.type", "custom");indexerSettings.put("analysis.analyzer.events.tokenizer", "standard");indexerSettings.put("analysis.analyzer.events.filter", "snowball, standard, lowercase");createRequest.setSettings(indexerSettings);return createRequest;

  1. What is the difference between the index name & index id? What is each of those used for?

Thanks!!

Ronak Patel


(Shay Banon) #18

It seems like you have events duplicated? I suggest you just do a simple
experiment (to simplify things), just index your data json into a fresh
index and execute get mappings on it, see the correct structure that you
need to provide elasticsearch with.

On Sun, Apr 15, 2012 at 7:52 PM, Ronak Patel ronak2121@gmail.com wrote:

I posted the mapping and settings when I created the index and it seems
that elasticsearch has accepted them.
So, that tells me that something is wrong with how elasticsearch is
picking up the template...

On Sun, Apr 15, 2012 at 12:28 PM, Ronak Patel ronak2121@gmail.com wrote:

The logs also keep mentioning the word source. Is that some json element
I need to have in this mapping?

On Sun, Apr 15, 2012 at 9:15 AM, Ronak Patel ronak2121@gmail.com wrote:

I put the admin config template in the /config/templates folder as
stated on the site. I assume that's the right spot?

On Sun, Apr 15, 2012 at 9:13 AM, Ronak Patel ronak2121@gmail.comwrote:

I do see in the logs that elasticsearch sets my settings so that part I
know works okay.
The mappings just don't seem to be getting set when I create a new
index called "events".

On Sat, Apr 14, 2012 at 12:40 AM, Ronak Patel ronak2121@gmail.comwrote:

Hi Shay,

Thanks a lot for helping me out.

I got this config file now:

{
"template": "events",
"settings": {
"analysis.analyzer.events.type": "custom",
"analysis.analyzer.events.tokenizer": "standard",
"analysis.analyzer.events.filter": "snowball, standard,
lowercase"
},
"mappings": {
"events": {
"events": {
"properties": {
"startDate": {
"format": "dateOptionalTime",
"type": "date"
},
"icon": {
"type": "string"
},
"phone": {
"type": "string"
},
"coordinate": {
"lat_lon": true,
"type": "geo_point"
},
"location": {
"type": "string"
},
"endDate": {
"format": "dateOptionalTime",
"type": "date"
},
"type": {
"type": "string"
},
"contact": {
"type": "string"
},
"url": {
"type": "string"
},
"costs": {
"type": "string"
},
"eventId": {
"type": "long"
},
"email": {
"type": "string"
},
"description": {
"type": "string"
},
"resortName": {
"type": "string"
},
"name": {
"type": "string"
},
"resortId": {
"type": "long"
}
}
}
}
}
}

Yet I still get errors stating that mappings for startDate could not
be found. This is happening while I am trying to query against an index
that's been created but contains no documents.

On Wed, Apr 11, 2012 at 5:30 AM, Shay Banon kimchy@gmail.com wrote:

It seems like your mappings are incorrect. Under an object level,
there should be "properties" element. So, if events is the type, and source
is the top level json object, both should have properties under them. Maybe
you can gist a sample json doc that shows the document you index, and then
helping with the mappings will be simpler. Also, you can use get mapping
API to see the automatic mapping created by elasticsearch for the data
indexed, and you can get a feeling for how it should look like.

On Mon, Apr 9, 2012 at 12:02 PM, Ronak Patel ronak2121@gmail.comwrote:

So I set up a template and I see elasticsearch picking it up in the
logs. However, I'm getting errors when I run queries against this index
about the "startDate" or "coordinate" field.

Those errors are:

Caused by: org.elasticsearch.index.query.QueryParsingException:
[events] failed to find geo_point field [coordinate]
at
org.elasticsearch.index.query.GeoDistanceFilterParser.parse(GeoDistanceFilterParser.java:180)
at
org.elasticsearch.index.query.QueryParseContext.parseInnerFilter(QueryParseContext.java:224)
at
org.elasticsearch.index.query.IndexQueryParserService.parseInnerFilter(IndexQueryParserService.java:232)
at
org.elasticsearch.search.query.FilterParseElement.parse(FilterParseElement.java:33)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:545)

Caused by: org.elasticsearch.search.SearchParseException:
[events][3]: query[type:club_trip],from[0],size[10]: Parse Failure [No
mapping found for [startDate] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:164)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:138)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:70)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:545)

Here is my template file:

{
"template" : {
"template" : "events",
"settings" : {
"analysis.analyzer.events.type" : "custom",
"analysis.analyzer.events.tokenizer" :
"standard",
"analysis.analyzer.events.filter" :
"snowball, standard, lowercase"
},
"mappings" : {
"events" : {
"source" : {
"eventId" : {
"type" : "int"
},
"coordinate" : {
"type" : "geo_point",
"lat_long" : true
},
"contact" : {
"type" : "string"
},
"costs" : {
"type" : "string"
},
"description" : {
"type" : "string"
},
"email" : {
"type" : "string"
},
"endDate" : {
"type" : "datetime"
},
"icon" : {
"type" : "string"
},
"location" : {
"type" : "string"
},
"name" : {
"type" : "string"
},
"phone" : {
"type" : "string"
},
"resortId" : {
"type" : "int"
},
"resortName" : {
"type" : "string"
},
"startDate" : {
"type" : "datetime"
},
"type" : {
"type" : "string"
},
"url" : {
"type" : "string"
}
}
}
}
}
}

This index does not contain any documents at the moment. But, I
would expect it to work fine as long as I had configured these mappings.

Is that not the case? (Or is there something wrong with my template
file?)

Thanks,

Ronak Patel

On Sun, Apr 8, 2012 at 2:24 PM, Shay Banon kimchy@gmail.com wrote:

If you want to provide an index level config, then you will need to
do it as part of an index template (with an example showed in my previous
link). The tempalte can simply match directly on the index name, and the
index level settings and mappings can eb set in the template.

On Sun, Apr 8, 2012 at 3:29 AM, Ronak Patel ronak2121@gmail.comwrote:

Hi Shay,

Would it be possible for you to provide a more complete example?
I'm looking to specify the index name, document type, the fields
in that document + the geospatial fields in YAML.

Can you please have a sample that shows how I can do all of those
things?

Thanks,

Ronak

On Tue, Mar 20, 2012 at 6:48 AM, Shay Banon kimchy@gmail.comwrote:

Yes, you can configure it in the elasticsearch.yml file (note, it
can also be a json one), or, you can configure in 0.19 index templates as
part of the configuration location (check at the bottom under Config):
http://www.elasticsearch.org/guide/reference/api/admin-indices-templates.html
.

On Mon, Mar 19, 2012 at 10:17 PM, Ronak Patel <
ronak2121@gmail.com> wrote:

I want to get rid of this code from the "client" side and move
it into the Elastic Search configuration. I saw that the YAML config file
has a way to specify this for ES but I couldn't find any good examples on
how to set it up.

On Fri, Mar 16, 2012 at 1:05 PM, Shay Banon kimchy@gmail.comwrote:

I don't really understand than what you after. The code you
posted created an index with its own specific settings from the "client"
side. Can you explicitly explain what you after?

On Thu, Mar 15, 2012 at 1:26 AM, Ronak Patel <
ronak2121@gmail.com> wrote:

Right so that configuration is still in my client app instead
of inside of ES?? That's not what I was looking for...

On Wed, Mar 14, 2012 at 8:19 AM, Shay Banon kimchy@gmail.comwrote:

You can load the relevant json form the classpath into a
string, and then provide it as a string to the Java API.

On Tuesday, March 13, 2012 at 10:25 PM, Ronak Patel wrote:

Thanks for your reply Shay.

Is this JSON something I would just paste into the config
file? My set of document types/field types are pretty static so I was
hoping to just configure it inside of ES and remove this extra
initialization code.

On Thu, Mar 1, 2012 at 7:28 AM, Shay Banon kimchy@gmail.comwrote:

Just build a json representing the settings, a json
representing the mapping for each type, and load it. You don't have to use
ES APis for it. Take your pick with whatever json library you wish to use.

On Wednesday, February 29, 2012 at 6:47 PM, Ronak Patel wrote:

Hi All,

I'm fairly new to ElasticSearch and am looking for the proper
way to configure it to create my index and my index settings instead of
relying on the application to do it using the ES API.

Is there any documentation or any snippits someone could
point me at to:

  1. Configure the Index Data Mappings and Create the Index

final CreateIndexRequestBuilder builder = this.
getCreateIndexRequestBuilder();

// create a mapping for a geocoordinatefinal Map<String, Object> source = new HashMap<String, Object>();final Map<String, Object> properties = new HashMap<String, Object>();final Map<String, Object> mapping = new HashMap<String, Object>();source.put("properties", properties);mapping.put("type", "geo_point");mapping.put("lat_lon", Boolean.TRUE);for (final String field : geoCoordinateFields) {

properties.put(field, mapping);

}builder.addMapping(this.documentType, source);// build the responsefinal CreateIndexResponse createResponse = builder.execute().get();if (createResponse.acknowledged()) {

DefaultSearchServiceImpl.LOGGER.info("Successfully created the index: {}", this.index);

} else {

DefaultSearchServiceImpl.LOGGER.error("Failed to create the index: {}", this.index);

}return createResponse.acknowledged();

  1. Handle Configuring the Lucene Analyzer
    final IndicesAdminClient indexService = this.searchService.
    admin().indices();

final CreateIndexRequestBuilder createRequest = indexService.prepareCreate(this.index);// set the default analyzersfinal ImmutableSettings.Builder indexerSettings = ImmutableSettings.settingsBuilder();indexerSettings.put("analysis.analyzer.events.type", "custom");indexerSettings.put("analysis.analyzer.events.tokenizer", "standard");indexerSettings.put("analysis.analyzer.events.filter", "snowball, standard, lowercase");createRequest.setSettings(indexerSettings);return createRequest;

  1. What is the difference between the index name & index id? What is each of those used for?

Thanks!!

Ronak Patel


(Ronak Patel) #19

Yea I was able to set the mappings just fine on index creation time using
that method. The problem I have was in making elasticsearch using the
config template. I couldn't see any logging statements where it mentioned
it was trying to apply the template.

On Tue, Apr 17, 2012 at 6:34 AM, Shay Banon kimchy@gmail.com wrote:

It seems like you have events duplicated? I suggest you just do a simple
experiment (to simplify things), just index your data json into a fresh
index and execute get mappings on it, see the correct structure that you
need to provide elasticsearch with.

On Sun, Apr 15, 2012 at 7:52 PM, Ronak Patel ronak2121@gmail.com wrote:

I posted the mapping and settings when I created the index and it seems
that elasticsearch has accepted them.
So, that tells me that something is wrong with how elasticsearch is
picking up the template...

On Sun, Apr 15, 2012 at 12:28 PM, Ronak Patel ronak2121@gmail.comwrote:

The logs also keep mentioning the word source. Is that some json element
I need to have in this mapping?

On Sun, Apr 15, 2012 at 9:15 AM, Ronak Patel ronak2121@gmail.comwrote:

I put the admin config template in the /config/templates folder as
stated on the site. I assume that's the right spot?

On Sun, Apr 15, 2012 at 9:13 AM, Ronak Patel ronak2121@gmail.comwrote:

I do see in the logs that elasticsearch sets my settings so that part
I know works okay.
The mappings just don't seem to be getting set when I create a new
index called "events".

On Sat, Apr 14, 2012 at 12:40 AM, Ronak Patel ronak2121@gmail.comwrote:

Hi Shay,

Thanks a lot for helping me out.

I got this config file now:

{
"template": "events",
"settings": {
"analysis.analyzer.events.type": "custom",
"analysis.analyzer.events.tokenizer": "standard",
"analysis.analyzer.events.filter": "snowball, standard,
lowercase"
},
"mappings": {
"events": {
"events": {
"properties": {
"startDate": {
"format": "dateOptionalTime",
"type": "date"
},
"icon": {
"type": "string"
},
"phone": {
"type": "string"
},
"coordinate": {
"lat_lon": true,
"type": "geo_point"
},
"location": {
"type": "string"
},
"endDate": {
"format": "dateOptionalTime",
"type": "date"
},
"type": {
"type": "string"
},
"contact": {
"type": "string"
},
"url": {
"type": "string"
},
"costs": {
"type": "string"
},
"eventId": {
"type": "long"
},
"email": {
"type": "string"
},
"description": {
"type": "string"
},
"resortName": {
"type": "string"
},
"name": {
"type": "string"
},
"resortId": {
"type": "long"
}
}
}
}
}
}

Yet I still get errors stating that mappings for startDate could not
be found. This is happening while I am trying to query against an index
that's been created but contains no documents.

On Wed, Apr 11, 2012 at 5:30 AM, Shay Banon kimchy@gmail.comwrote:

It seems like your mappings are incorrect. Under an object level,
there should be "properties" element. So, if events is the type, and source
is the top level json object, both should have properties under them. Maybe
you can gist a sample json doc that shows the document you index, and then
helping with the mappings will be simpler. Also, you can use get mapping
API to see the automatic mapping created by elasticsearch for the data
indexed, and you can get a feeling for how it should look like.

On Mon, Apr 9, 2012 at 12:02 PM, Ronak Patel ronak2121@gmail.comwrote:

So I set up a template and I see elasticsearch picking it up in the
logs. However, I'm getting errors when I run queries against this index
about the "startDate" or "coordinate" field.

Those errors are:

Caused by: org.elasticsearch.index.query.QueryParsingException:
[events] failed to find geo_point field [coordinate]
at
org.elasticsearch.index.query.GeoDistanceFilterParser.parse(GeoDistanceFilterParser.java:180)
at
org.elasticsearch.index.query.QueryParseContext.parseInnerFilter(QueryParseContext.java:224)
at
org.elasticsearch.index.query.IndexQueryParserService.parseInnerFilter(IndexQueryParserService.java:232)
at
org.elasticsearch.search.query.FilterParseElement.parse(FilterParseElement.java:33)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:545)

Caused by: org.elasticsearch.search.SearchParseException:
[events][3]: query[type:club_trip],from[0],size[10]: Parse Failure [No
mapping found for [startDate] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:164)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:138)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:70)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:545)

Here is my template file:

{
"template" : {
"template" : "events",
"settings" : {
"analysis.analyzer.events.type" : "custom",
"analysis.analyzer.events.tokenizer" :
"standard",
"analysis.analyzer.events.filter" :
"snowball, standard, lowercase"
},
"mappings" : {
"events" : {
"source" : {
"eventId" : {
"type" : "int"
},
"coordinate" : {
"type" :
"geo_point",
"lat_long" : true
},
"contact" : {
"type" : "string"
},
"costs" : {
"type" : "string"
},
"description" : {
"type" : "string"
},
"email" : {
"type" : "string"
},
"endDate" : {
"type" : "datetime"
},
"icon" : {
"type" : "string"
},
"location" : {
"type" : "string"
},
"name" : {
"type" : "string"
},
"phone" : {
"type" : "string"
},
"resortId" : {
"type" : "int"
},
"resortName" : {
"type" : "string"
},
"startDate" : {
"type" : "datetime"
},
"type" : {
"type" : "string"
},
"url" : {
"type" : "string"
}
}
}
}
}
}

This index does not contain any documents at the moment. But, I
would expect it to work fine as long as I had configured these mappings.

Is that not the case? (Or is there something wrong with my template
file?)

Thanks,

Ronak Patel

On Sun, Apr 8, 2012 at 2:24 PM, Shay Banon kimchy@gmail.comwrote:

If you want to provide an index level config, then you will need
to do it as part of an index template (with an example showed in my
previous link). The tempalte can simply match directly on the index name,
and the index level settings and mappings can eb set in the template.

On Sun, Apr 8, 2012 at 3:29 AM, Ronak Patel ronak2121@gmail.comwrote:

Hi Shay,

Would it be possible for you to provide a more complete example?
I'm looking to specify the index name, document type, the fields
in that document + the geospatial fields in YAML.

Can you please have a sample that shows how I can do all of those
things?

Thanks,

Ronak

On Tue, Mar 20, 2012 at 6:48 AM, Shay Banon kimchy@gmail.comwrote:

Yes, you can configure it in the elasticsearch.yml file (note,
it can also be a json one), or, you can configure in 0.19 index templates
as part of the configuration location (check at the bottom under Config):
http://www.elasticsearch.org/guide/reference/api/admin-indices-templates.html
.

On Mon, Mar 19, 2012 at 10:17 PM, Ronak Patel <
ronak2121@gmail.com> wrote:

I want to get rid of this code from the "client" side and move
it into the Elastic Search configuration. I saw that the YAML config file
has a way to specify this for ES but I couldn't find any good examples on
how to set it up.

On Fri, Mar 16, 2012 at 1:05 PM, Shay Banon kimchy@gmail.comwrote:

I don't really understand than what you after. The code you
posted created an index with its own specific settings from the "client"
side. Can you explicitly explain what you after?

On Thu, Mar 15, 2012 at 1:26 AM, Ronak Patel <
ronak2121@gmail.com> wrote:

Right so that configuration is still in my client app instead
of inside of ES?? That's not what I was looking for...

On Wed, Mar 14, 2012 at 8:19 AM, Shay Banon <kimchy@gmail.com

wrote:

You can load the relevant json form the classpath into a
string, and then provide it as a string to the Java API.

On Tuesday, March 13, 2012 at 10:25 PM, Ronak Patel wrote:

Thanks for your reply Shay.

Is this JSON something I would just paste into the config
file? My set of document types/field types are pretty static so I was
hoping to just configure it inside of ES and remove this extra
initialization code.

On Thu, Mar 1, 2012 at 7:28 AM, Shay Banon <kimchy@gmail.com

wrote:

Just build a json representing the settings, a json
representing the mapping for each type, and load it. You don't have to use
ES APis for it. Take your pick with whatever json library you wish to use.

On Wednesday, February 29, 2012 at 6:47 PM, Ronak Patel
wrote:

Hi All,

I'm fairly new to ElasticSearch and am looking for the
proper way to configure it to create my index and my index settings instead
of relying on the application to do it using the ES API.

Is there any documentation or any snippits someone could
point me at to:

  1. Configure the Index Data Mappings and Create the Index

final CreateIndexRequestBuilder builder = this.
getCreateIndexRequestBuilder();

// create a mapping for a geocoordinatefinal Map<String, Object> source = new HashMap<String, Object>();final Map<String, Object> properties = new HashMap<String, Object>();final Map<String, Object> mapping = new HashMap<String, Object>();source.put("properties", properties);mapping.put("type", "geo_point");mapping.put("lat_lon", Boolean.TRUE);for (final String field : geoCoordinateFields) {

properties.put(field, mapping);

}builder.addMapping(this.documentType, source);// build the responsefinal CreateIndexResponse createResponse = builder.execute().get();if (createResponse.acknowledged()) {

DefaultSearchServiceImpl.LOGGER.info("Successfully created the index: {}", this.index);

} else {

DefaultSearchServiceImpl.LOGGER.error("Failed to create the index: {}", this.index);

}return createResponse.acknowledged();

  1. Handle Configuring the Lucene Analyzer
    final IndicesAdminClient indexService = this.searchService.
    admin().indices();

final CreateIndexRequestBuilder createRequest = indexService.prepareCreate(this.index);// set the default analyzersfinal ImmutableSettings.Builder indexerSettings = ImmutableSettings.settingsBuilder();indexerSettings.put("analysis.analyzer.events.type", "custom");indexerSettings.put("analysis.analyzer.events.tokenizer", "standard");indexerSettings.put("analysis.analyzer.events.filter", "snowball, standard, lowercase");createRequest.setSettings(indexerSettings);return createRequest;

  1. What is the difference between the index name & index id? What is each of those used for?

Thanks!!

Ronak Patel


(system) #20