Flexible index mappings


(Ivan Brusic) #1

In my current setup, I am configuring new index mappings via a file on
each node. File-based mappings allow me to change items at the
application level and not via code, and are easier to manage in
source-control.

However, with using index aliases, the only way I can apply to same
mapping to each new index is by putting the file in the _default
directory. Currently the system is only using one index, so the
solution is acceptable, but other indices will appear in the future.

Having a more flexible approach is needed, but I am unsure about any
practical suggestions. Having wildcards in the directory name is ugly
and not supported by Windows. Perhaps a default.json in the
config/mappings directory with a slightly different JSON structure to
allow index names (with wildcards)?

e.g.

{
"myindex-*" : {
"mytype" : {
"_source" : {"enabled" : true},
"_all" : {"enabled" : true},
"properties" : {
...
This file would have the lowest precedence in determining the correct mapping.

Or perhaps am I missing another solution altogether?

Cheers,

Ivan


(Karussell) #2

Or perhaps am I missing another solution altogether?

In ElasticSearch you define per type

http://www.elasticsearch.org/guide/reference/mapping/object-type.html

Or did I misunderstand something?

Regards,
Peter.


(Ivan Brusic) #3

There are only two options for placing the mapping file: inside the
_default directory which applies to all indices, or inside a directory
with the same name as an index. Types defined inside a mapping inside
that _default will apply to all indices, even if the type is not used.
No performance issues, but the index metadata will have this mapping,
which can be confusing.

On Thu, Feb 9, 2012 at 1:56 AM, Karussell tableyourtime@googlemail.com wrote:

Or perhaps am I missing another solution altogether?

In ElasticSearch you define per type

http://www.elasticsearch.org/guide/reference/mapping/object-type.html

Or did I misunderstand something?

Regards,
Peter.


(Shay Banon) #4

How about using index templates? You can't place index templates in the configuration location of node(s), but you can place them when the system starts, and then they will apply to indices matching the relevant index name.

On Thursday, February 9, 2012 at 6:03 PM, Ivan Brusic wrote:

There are only two options for placing the mapping file: inside the
_default directory which applies to all indices, or inside a directory
with the same name as an index. Types defined inside a mapping inside
that _default will apply to all indices, even if the type is not used.
No performance issues, but the index metadata will have this mapping,
which can be confusing.

On Thu, Feb 9, 2012 at 1:56 AM, Karussell <tableyourtime@googlemail.com (mailto:tableyourtime@googlemail.com)> wrote:

Or perhaps am I missing another solution altogether?

In ElasticSearch you define per type

http://www.elasticsearch.org/guide/reference/mapping/object-type.html

Or did I misunderstand something?

Regards,
Peter.


(Ivan Brusic) #5

I did consider index templates since they allow the use of wildcards
in the index name, but did not explore further since they are not file
based.

Not sure what you mean by "you can place them when the system starts".
Do you mean they can be added via the API at startup?

Ultimately, the file based approach might not work since they require a reboot.

--
Ivan

On Sun, Feb 12, 2012 at 3:20 AM, Shay Banon kimchy@gmail.com wrote:

How about using index templates? You can't place index templates in the
configuration location of node(s), but you can place them when the system
starts, and then they will apply to indices matching the relevant index
name.

On Thursday, February 9, 2012 at 6:03 PM, Ivan Brusic wrote:

There are only two options for placing the mapping file: inside the
_default directory which applies to all indices, or inside a directory
with the same name as an index. Types defined inside a mapping inside
that _default will apply to all indices, even if the type is not used.
No performance issues, but the index metadata will have this mapping,
which can be confusing.

On Thu, Feb 9, 2012 at 1:56 AM, Karussell tableyourtime@googlemail.com
wrote:

Or perhaps am I missing another solution altogether?

In ElasticSearch you define per type

http://www.elasticsearch.org/guide/reference/mapping/object-type.html

Or did I misunderstand something?

Regards,
Peter.


(Shay Banon) #6

I added this so you can place templates under config: https://github.com/elasticsearch/elasticsearch/issues/1703.

On Wednesday, February 15, 2012 at 12:56 AM, Ivan Brusic wrote:

I did consider index templates since they allow the use of wildcards
in the index name, but did not explore further since they are not file
based.

Not sure what you mean by "you can place them when the system starts".
Do you mean they can be added via the API at startup?

Ultimately, the file based approach might not work since they require a reboot.

--
Ivan

On Sun, Feb 12, 2012 at 3:20 AM, Shay Banon <kimchy@gmail.com (mailto:kimchy@gmail.com)> wrote:

How about using index templates? You can't place index templates in the
configuration location of node(s), but you can place them when the system
starts, and then they will apply to indices matching the relevant index
name.

On Thursday, February 9, 2012 at 6:03 PM, Ivan Brusic wrote:

There are only two options for placing the mapping file: inside the
_default directory which applies to all indices, or inside a directory
with the same name as an index. Types defined inside a mapping inside
that _default will apply to all indices, even if the type is not used.
No performance issues, but the index metadata will have this mapping,
which can be confusing.

On Thu, Feb 9, 2012 at 1:56 AM, Karussell <tableyourtime@googlemail.com (mailto:tableyourtime@googlemail.com)>
wrote:

Or perhaps am I missing another solution altogether?

In ElasticSearch you define per type

http://www.elasticsearch.org/guide/reference/mapping/object-type.html

Or did I misunderstand something?

Regards,
Peter.


(Ivan Brusic) #7

Finally had a chance to test out the change with 0.19.0 (maintenance
coding is taking up all my time). Works like a charm, but any
modifications to a template are not reflected in the index, even after
a restart. Is this an issue with dynamic templates (which I have never
used) in general?

--
Ivan

On Wed, Feb 15, 2012 at 2:36 AM, Shay Banon kimchy@gmail.com wrote:

I added this so you can place templates under
config: https://github.com/elasticsearch/elasticsearch/issues/1703.

On Wednesday, February 15, 2012 at 12:56 AM, Ivan Brusic wrote:

I did consider index templates since they allow the use of wildcards
in the index name, but did not explore further since they are not file
based.

Not sure what you mean by "you can place them when the system starts".
Do you mean they can be added via the API at startup?

Ultimately, the file based approach might not work since they require a
reboot.

--
Ivan

On Sun, Feb 12, 2012 at 3:20 AM, Shay Banon kimchy@gmail.com wrote:

How about using index templates? You can't place index templates in the
configuration location of node(s), but you can place them when the system
starts, and then they will apply to indices matching the relevant index
name.

On Thursday, February 9, 2012 at 6:03 PM, Ivan Brusic wrote:

There are only two options for placing the mapping file: inside the
_default directory which applies to all indices, or inside a directory
with the same name as an index. Types defined inside a mapping inside
that _default will apply to all indices, even if the type is not used.
No performance issues, but the index metadata will have this mapping,
which can be confusing.

On Thu, Feb 9, 2012 at 1:56 AM, Karussell tableyourtime@googlemail.com
wrote:

Or perhaps am I missing another solution altogether?

In ElasticSearch you define per type

http://www.elasticsearch.org/guide/reference/mapping/object-type.html

Or did I misunderstand something?

Regards,
Peter.


(Radu Gheorghe) #8

Hi Ivan,

I know it's a long shot, but are you sure you actually modified the
template, and not entered a new one without deleting the old template?
That's what I did yesterday, so I thought I should share :smiley:

To get all your templates, you can do something like this:

curl -XGET 'http://localhost:9200/_cluster/state?pretty=true' > /tmp/

cluster_state_dump

and then look in that file for "templates".

Best regards,
Radu

On Mar 14, 11:21 pm, Ivan Brusic i...@brusic.com wrote:

Finally had a chance to test out the change with 0.19.0 (maintenance
coding is taking up all my time). Works like a charm, but any
modifications to a template are not reflected in the index, even after
a restart. Is this an issue with dynamic templates (which I have never
used) in general?

--
Ivan

On Wed, Feb 15, 2012 at 2:36 AM, Shay Banon kim...@gmail.com wrote:

I added this so you can place templates under
config: https://github.com/elasticsearch/elasticsearch/issues/1703.

On Wednesday, February 15, 2012 at 12:56 AM, Ivan Brusic wrote:

I did consider index templates since they allow the use of wildcards
in the index name, but did not explore further since they are not file
based.

Not sure what you mean by "you can place them when the system starts".
Do you mean they can be added via the API at startup?

Ultimately, the file based approach might not work since they require a
reboot.

--
Ivan

On Sun, Feb 12, 2012 at 3:20 AM, Shay Banon kim...@gmail.com wrote:

How about using index templates? You can't place index templates in the
configuration location of node(s), but you can place them when the system
starts, and then they will apply to indices matching the relevant index
name.

On Thursday, February 9, 2012 at 6:03 PM, Ivan Brusic wrote:

There are only two options for placing the mapping file: inside the
_default directory which applies to all indices, or inside a directory
with the same name as an index. Types defined inside a mapping inside
that _default will apply to all indices, even if the type is not used.
No performance issues, but the index metadata will have this mapping,
which can be confusing.

On Thu, Feb 9, 2012 at 1:56 AM, Karussell tableyourt...@googlemail.com
wrote:

Or perhaps am I missing another solution altogether?

In ElasticSearch you define per type

http://www.elasticsearch.org/guide/reference/mapping/object-type.html

Or did I misunderstand something?

Regards,
Peter.


(Shay Banon) #9

Modifying the template will only apply to new indices created, not to
existing indices...

On Wed, Mar 14, 2012 at 11:21 PM, Ivan Brusic ivan@brusic.com wrote:

Finally had a chance to test out the change with 0.19.0 (maintenance
coding is taking up all my time). Works like a charm, but any
modifications to a template are not reflected in the index, even after
a restart. Is this an issue with dynamic templates (which I have never
used) in general?

--
Ivan

On Wed, Feb 15, 2012 at 2:36 AM, Shay Banon kimchy@gmail.com wrote:

I added this so you can place templates under
config: https://github.com/elasticsearch/elasticsearch/issues/1703.

On Wednesday, February 15, 2012 at 12:56 AM, Ivan Brusic wrote:

I did consider index templates since they allow the use of wildcards
in the index name, but did not explore further since they are not file
based.

Not sure what you mean by "you can place them when the system starts".
Do you mean they can be added via the API at startup?

Ultimately, the file based approach might not work since they require a
reboot.

--
Ivan

On Sun, Feb 12, 2012 at 3:20 AM, Shay Banon kimchy@gmail.com wrote:

How about using index templates? You can't place index templates in the
configuration location of node(s), but you can place them when the system
starts, and then they will apply to indices matching the relevant index
name.

On Thursday, February 9, 2012 at 6:03 PM, Ivan Brusic wrote:

There are only two options for placing the mapping file: inside the
_default directory which applies to all indices, or inside a directory
with the same name as an index. Types defined inside a mapping inside
that _default will apply to all indices, even if the type is not used.
No performance issues, but the index metadata will have this mapping,
which can be confusing.

On Thu, Feb 9, 2012 at 1:56 AM, Karussell tableyourtime@googlemail.com
wrote:

Or perhaps am I missing another solution altogether?

In ElasticSearch you define per type

http://www.elasticsearch.org/guide/reference/mapping/object-type.html

Or did I misunderstand something?

Regards,
Peter.


(system) #10