Proper YAML Configuration of ElasticSearch

Elastic Stack Elasticsearch
14

I do see in the logs that elasticsearch sets my settings so that part I
know works okay.
The mappings just don't seem to be getting set when I create a new index
called "events".

On Sat, Apr 14, 2012 at 12:40 AM, Ronak Patel ronak2121@gmail.com wrote:

Hi Shay,

Thanks a lot for helping me out.

I got this config file now:

{
"template": "events",
"settings": {
"analysis.analyzer.events.type": "custom",
"analysis.analyzer.events.tokenizer": "standard",
"analysis.analyzer.events.filter": "snowball, standard, lowercase"
},
"mappings": {
"events": {
"events": {
"properties": {
"startDate": {
"format": "dateOptionalTime",
"type": "date"
},
"icon": {
"type": "string"
},
"phone": {
"type": "string"
},
"coordinate": {
"lat_lon": true,
"type": "geo_point"
},
"location": {
"type": "string"
},
"endDate": {
"format": "dateOptionalTime",
"type": "date"
},
"type": {
"type": "string"
},
"contact": {
"type": "string"
},
"url": {
"type": "string"
},
"costs": {
"type": "string"
},
"eventId": {
"type": "long"
},
"email": {
"type": "string"
},
"description": {
"type": "string"
},
"resortName": {
"type": "string"
},
"name": {
"type": "string"
},
"resortId": {
"type": "long"
}
}
}
}
}
}

Yet I still get errors stating that mappings for startDate could not be
found. This is happening while I am trying to query against an index that's
been created but contains no documents.

On Wed, Apr 11, 2012 at 5:30 AM, Shay Banon kimchy@gmail.com wrote:

It seems like your mappings are incorrect. Under an object level, there
should be "properties" element. So, if events is the type, and source is
the top level json object, both should have properties under them. Maybe
you can gist a sample json doc that shows the document you index, and then
helping with the mappings will be simpler. Also, you can use get mapping
API to see the automatic mapping created by elasticsearch for the data
indexed, and you can get a feeling for how it should look like.

On Mon, Apr 9, 2012 at 12:02 PM, Ronak Patel ronak2121@gmail.com wrote:

So I set up a template and I see elasticsearch picking it up in the
logs. However, I'm getting errors when I run queries against this index
about the "startDate" or "coordinate" field.

Those errors are:

Caused by: org.elasticsearch.index.query.QueryParsingException: [events]
failed to find geo_point field [coordinate]
at
org.elasticsearch.index.query.GeoDistanceFilterParser.parse(GeoDistanceFilterParser.java:180)
at
org.elasticsearch.index.query.QueryParseContext.parseInnerFilter(QueryParseContext.java:224)
at
org.elasticsearch.index.query.IndexQueryParserService.parseInnerFilter(IndexQueryParserService.java:232)
at
org.elasticsearch.search.query.FilterParseElement.parse(FilterParseElement.java:33)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:545)

Caused by: org.elasticsearch.search.SearchParseException: [events][3]:
query[type:club_trip],from[0],size[10]: Parse Failure [No mapping found for
[startDate] in order to sort on]
at
org.elasticsearch.search.sort.SortParseElement.addSortField(SortParseElement.java:164)
at
org.elasticsearch.search.sort.SortParseElement.addCompoundSortField(SortParseElement.java:138)
at
org.elasticsearch.search.sort.SortParseElement.parse(SortParseElement.java:70)
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:545)

Here is my template file:

{
"template" : {
"template" : "events",
"settings" : {
"analysis.analyzer.events.type" : "custom",
"analysis.analyzer.events.tokenizer" :
"standard",
"analysis.analyzer.events.filter" : "snowball,
standard, lowercase"
},
"mappings" : {
"events" : {
"source" : {
"eventId" : {
"type" : "int"
},
"coordinate" : {
"type" : "geo_point",
"lat_long" : true
},
"contact" : {
"type" : "string"
},
"costs" : {
"type" : "string"
},
"description" : {
"type" : "string"
},
"email" : {
"type" : "string"
},
"endDate" : {
"type" : "datetime"
},
"icon" : {
"type" : "string"
},
"location" : {
"type" : "string"
},
"name" : {
"type" : "string"
},
"phone" : {
"type" : "string"
},
"resortId" : {
"type" : "int"
},
"resortName" : {
"type" : "string"
},
"startDate" : {
"type" : "datetime"
},
"type" : {
"type" : "string"
},
"url" : {
"type" : "string"
}
}
}
}
}
}

This index does not contain any documents at the moment. But, I would
expect it to work fine as long as I had configured these mappings.

Is that not the case? (Or is there something wrong with my template
file?)

Thanks,

Ronak Patel

On Sun, Apr 8, 2012 at 2:24 PM, Shay Banon kimchy@gmail.com wrote:

If you want to provide an index level config, then you will need to do
it as part of an index template (with an example showed in my previous
link). The tempalte can simply match directly on the index name, and the
index level settings and mappings can eb set in the template.

On Sun, Apr 8, 2012 at 3:29 AM, Ronak Patel ronak2121@gmail.comwrote:

Hi Shay,

Would it be possible for you to provide a more complete example?
I'm looking to specify the index name, document type, the fields in
that document + the geospatial fields in YAML.

Can you please have a sample that shows how I can do all of those
things?

Thanks,

Ronak

On Tue, Mar 20, 2012 at 6:48 AM, Shay Banon kimchy@gmail.com wrote:

Yes, you can configure it in the elasticsearch.yml file (note, it can
also be a json one), or, you can configure in 0.19 index templates as part
of the configuration location (check at the bottom under Config):
Elasticsearch Platform — Find real-time answers at scale | Elastic
.

On Mon, Mar 19, 2012 at 10:17 PM, Ronak Patel ronak2121@gmail.comwrote:

I want to get rid of this code from the "client" side and move it
into the Elastic Search configuration. I saw that the YAML config file has
a way to specify this for ES but I couldn't find any good examples on how
to set it up.

On Fri, Mar 16, 2012 at 1:05 PM, Shay Banon kimchy@gmail.comwrote:

I don't really understand than what you after. The code you posted
created an index with its own specific settings from the "client" side. Can
you explicitly explain what you after?

On Thu, Mar 15, 2012 at 1:26 AM, Ronak Patel ronak2121@gmail.comwrote:

Right so that configuration is still in my client app instead of
inside of ES?? That's not what I was looking for...

On Wed, Mar 14, 2012 at 8:19 AM, Shay Banon kimchy@gmail.comwrote:

You can load the relevant json form the classpath into a string,
and then provide it as a string to the Java API.

On Tuesday, March 13, 2012 at 10:25 PM, Ronak Patel wrote:

Thanks for your reply Shay.

Is this JSON something I would just paste into the config file?
My set of document types/field types are pretty static so I was hoping to
just configure it inside of ES and remove this extra initialization code.

On Thu, Mar 1, 2012 at 7:28 AM, Shay Banon kimchy@gmail.comwrote:

Just build a json representing the settings, a json representing
the mapping for each type, and load it. You don't have to use ES APis for
it. Take your pick with whatever json library you wish to use.

On Wednesday, February 29, 2012 at 6:47 PM, Ronak Patel wrote:

Hi All,

I'm fairly new to Elasticsearch and am looking for the proper way
to configure it to create my index and my index settings instead of relying
on the application to do it using the ES API.

Is there any documentation or any snippits someone could point me
at to:

  1. Configure the Index Data Mappings and Create the Index

final CreateIndexRequestBuilder builder = this.
getCreateIndexRequestBuilder();

// create a mapping for a geocoordinatefinal Map<String, Object> source = new HashMap<String, Object>();final Map<String, Object> properties = new HashMap<String, Object>();final Map<String, Object> mapping = new HashMap<String, Object>();source.put("properties", properties);mapping.put("type", "geo_point");mapping.put("lat_lon", Boolean.TRUE);for (final String field : geoCoordinateFields) {

properties.put(field, mapping);

}builder.addMapping(this.documentType, source);// build the responsefinal CreateIndexResponse createResponse = builder.execute().get();if (createResponse.acknowledged()) {

DefaultSearchServiceImpl.LOGGER.info("Successfully created the index: {}", this.index);

} else {

DefaultSearchServiceImpl.LOGGER.error("Failed to create the index: {}", this.index);

}return createResponse.acknowledged();

  1. Handle Configuring the Lucene Analyzer
    final IndicesAdminClient indexService = this.searchService.admin
    ().indices();

final CreateIndexRequestBuilder createRequest = indexService.prepareCreate(this.index);// set the default analyzersfinal ImmutableSettings.Builder indexerSettings = ImmutableSettings.settingsBuilder();indexerSettings.put("analysis.analyzer.events.type", "custom");indexerSettings.put("analysis.analyzer.events.tokenizer", "standard");indexerSettings.put("analysis.analyzer.events.filter", "snowball, standard, lowercase");createRequest.setSettings(indexerSettings);return createRequest;

  1. What is the difference between the index name & index id? What is each of those used for?

Thanks!!

Ronak Patel