Daily index with static mapping


(Mohamed TEYEB) #1

Hi,

Right now, I have only one index which does contain 1 type. So I did create
the mapping and place under the config/mapping as explained by the
documentation :
http://www.elastic.co/guide/en/elasticsearch/reference/0.90/mapping-conf-mappings.html.

For the future, I would like to do the following :

  • Index will be daily : index_name_YYYYMMDD
  • Index will contains N types : index_name_type1, index_name_type2, ...
    index_name_typeN
  • All types will have the same format/mapping.

How can I handle this case with a pre-configured mapping set in the
sub-folder config/mapping ?
I know that I can handle it somehow but putting a default mapping, but will
it work with more than one index case?
Example : I would do a default mapping for all indexes which have a
common prefix (INDEX_A_YYYYMMDD => mapping_A, INDEX_B_YYYYMMDD =>
mapping_B, etc ...)

BR/

--
Please update your bookmarks! We moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a06c0542-1910-4fc7-8a3f-ca899e36fc4d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #2

You want to look at
http://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html
Also, the link you provided earlier is from 0.90.X, which is now really
old. If you are on that version it's best to upgrade.

PS - We're moving to https://discuss.elastic.co/, please join us there for
any future discussions!

On 11 May 2015 at 06:49, Mohamed TEYEB mohamed.teyeb@gmail.com wrote:

Hi,

Right now, I have only one index which does contain 1 type. So I did
create the mapping and place under the config/mapping as explained by the
documentation :
http://www.elastic.co/guide/en/elasticsearch/reference/0.90/mapping-conf-mappings.html
.

For the future, I would like to do the following :

  • Index will be daily : index_name_YYYYMMDD
  • Index will contains N types : index_name_type1, index_name_type2,
    ... index_name_typeN
  • All types will have the same format/mapping.

How can I handle this case with a pre-configured mapping set in the
sub-folder config/mapping ?
I know that I can handle it somehow but putting a default mapping, but
will it work with more than one index case?
Example : I would do a default mapping for all indexes which have a
common prefix (INDEX_A_YYYYMMDD => mapping_A, INDEX_B_YYYYMMDD =>
mapping_B, etc ...)

BR/

--
Please update your bookmarks! We moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a06c0542-1910-4fc7-8a3f-ca899e36fc4d%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/a06c0542-1910-4fc7-8a3f-ca899e36fc4d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9x_4Rhn-tEkwaqG6xYYXa951349zUk1a3nMw5tZ8Nqrg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Mohamed TEYEB) #3

Hi,

Thanks for you answer. Following the instructions above, I have created 1
template to handle a daily based index with multi types. I created a json
file which I dropped under config/templates.
file name is tpl-INDEX_NAME.json
content :

{
"tpl-INDEX_NAME" : {
"template" : "INDEX_NAME-*",
"settings" : {
"number_of_shards" : 5,
"number_of_replicas": 0
},
"mappings" : {
"default" : {
"properties" : {
....

}

but when I post a document with INDEX_NAME-20150514, INDEX_TYPE1. Elastic
search uses a dynamic mapping :

[2015-05-14 18:49:32,395][INFO ][cluster.metadata ] [host] [
INDEX_NAME-20150514] update_mapping [INDEX_TYPE1] (dynamic)

Any idea about what could be the missing part ?

BR.

On Sunday, 10 May 2015 23:37:06 UTC+2, Mark Walkom wrote:

You want to look at
http://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html
Also, the link you provided earlier is from 0.90.X, which is now really
old. If you are on that version it's best to upgrade.

PS - We're moving to https://discuss.elastic.co/, please join us there
for any future discussions!

On 11 May 2015 at 06:49, Mohamed TEYEB <mohame...@gmail.com <javascript:>>
wrote:

Hi,

Right now, I have only one index which does contain 1 type. So I did
create the mapping and place under the config/mapping as explained by the
documentation :
http://www.elastic.co/guide/en/elasticsearch/reference/0.90/mapping-conf-mappings.html
.

For the future, I would like to do the following :

  • Index will be daily : index_name_YYYYMMDD
  • Index will contains N types : index_name_type1, index_name_type2,
    ... index_name_typeN
  • All types will have the same format/mapping.

How can I handle this case with a pre-configured mapping set in the
sub-folder config/mapping ?
I know that I can handle it somehow but putting a default mapping, but
will it work with more than one index case?
Example : I would do a default mapping for all indexes which have a
common prefix (INDEX_A_YYYYMMDD => mapping_A, INDEX_B_YYYYMMDD =>
mapping_B, etc ...)

BR/

--
Please update your bookmarks! We moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a06c0542-1910-4fc7-8a3f-ca899e36fc4d%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/a06c0542-1910-4fc7-8a3f-ca899e36fc4d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/182cdd0c-2f15-4f23-aeaa-be184739922a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #4

That just means that the data in a document/event that was sent to ES did
not match the mapping that you provided. Buy default, ES will dynamically
map any new fields and then add them to the existing mapping.

To check what that difference is, just grab the mapping from ES and then
compare to the one in the file.

PS - We're moving to https://discuss.elastic.co/, please join us there for
any future discussions!

On 15 May 2015 at 03:02, Mohamed T. mohamed.teyeb@gmail.com wrote:

Hi,

Thanks for you answer. Following the instructions above, I have created 1
template to handle a daily based index with multi types. I created a json
file which I dropped under config/templates.
file name is tpl-INDEX_NAME.json
content :

{
"tpl-INDEX_NAME" : {
"template" : "INDEX_NAME-*",
"settings" : {
"number_of_shards" : 5,
"number_of_replicas": 0
},
"mappings" : {
"default" : {
"properties" : {
....

}

but when I post a document with INDEX_NAME-20150514, INDEX_TYPE1. Elastic
search uses a dynamic mapping :

[2015-05-14 18:49:32,395][INFO ][cluster.metadata ] [host] [
INDEX_NAME-20150514] update_mapping [INDEX_TYPE1] (dynamic)

Any idea about what could be the missing part ?

BR.

On Sunday, 10 May 2015 23:37:06 UTC+2, Mark Walkom wrote:

You want to look at
http://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html
Also, the link you provided earlier is from 0.90.X, which is now really
old. If you are on that version it's best to upgrade.

PS - We're moving to https://discuss.elastic.co/, please join us there
for any future discussions!

On 11 May 2015 at 06:49, Mohamed TEYEB mohame...@gmail.com wrote:

Hi,

Right now, I have only one index which does contain 1 type. So I did
create the mapping and place under the config/mapping as explained by the
documentation :
http://www.elastic.co/guide/en/elasticsearch/reference/0.90/mapping-conf-mappings.html
.

For the future, I would like to do the following :

  • Index will be daily : index_name_YYYYMMDD
  • Index will contains N types : index_name_type1, index_name_type2,
    ... index_name_typeN
  • All types will have the same format/mapping.

How can I handle this case with a pre-configured mapping set in the
sub-folder config/mapping ?
I know that I can handle it somehow but putting a default mapping, but
will it work with more than one index case?
Example : I would do a default mapping for all indexes which have a
common prefix (INDEX_A_YYYYMMDD => mapping_A, INDEX_B_YYYYMMDD =>
mapping_B, etc ...)

BR/

--
Please update your bookmarks! We moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a06c0542-1910-4fc7-8a3f-ca899e36fc4d%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/a06c0542-1910-4fc7-8a3f-ca899e36fc4d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/


You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/182cdd0c-2f15-4f23-aeaa-be184739922a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/182cdd0c-2f15-4f23-aeaa-be184739922a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X97Kry7tW12cTcD0OzkasEDofhnM0o2LsXbnbvS-dFVrA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Mohamed TEYEB) #5

Hi Mark,

I checked and the only difference is the name of the type ... but also a
type called 'default' get created and nothing common with the mapping
that I have set-up in the file.

I have also another question about my use case, my types will be handling
the language of the documents basically and I would like to know if it's
possible to fix a different analyzer and warmers for types linked to the
same index and not the index itself ? (the language code will be the last
2/3 chars of each type and also a field of each document)

BR.

On Thursday, 14 May 2015 23:49:16 UTC+2, Mark Walkom wrote:

That just means that the data in a document/event that was sent to ES did
not match the mapping that you provided. Buy default, ES will dynamically
map any new fields and then add them to the existing mapping.

To check what that difference is, just grab the mapping from ES and then
compare to the one in the file.

PS - We're moving to https://discuss.elastic.co/, please join us there
for any future discussions!

On 15 May 2015 at 03:02, Mohamed T. <mohame...@gmail.com <javascript:>>
wrote:

Hi,

Thanks for you answer. Following the instructions above, I have created 1
template to handle a daily based index with multi types. I created a json
file which I dropped under config/templates.
file name is tpl-INDEX_NAME.json
content :

{
"tpl-INDEX_NAME" : {
"template" : "INDEX_NAME-*",
"settings" : {
"number_of_shards" : 5,
"number_of_replicas": 0
},
"mappings" : {
"default" : {
"properties" : {
....

}

but when I post a document with INDEX_NAME-20150514, INDEX_TYPE1. Elastic
search uses a dynamic mapping :

[2015-05-14 18:49:32,395][INFO ][cluster.metadata ] [host] [
INDEX_NAME-20150514] update_mapping [INDEX_TYPE1] (dynamic)

Any idea about what could be the missing part ?

BR.

On Sunday, 10 May 2015 23:37:06 UTC+2, Mark Walkom wrote:

You want to look at
http://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html
Also, the link you provided earlier is from 0.90.X, which is now really
old. If you are on that version it's best to upgrade.

PS - We're moving to https://discuss.elastic.co/, please join us there
for any future discussions!

On 11 May 2015 at 06:49, Mohamed TEYEB mohame...@gmail.com wrote:

Hi,

Right now, I have only one index which does contain 1 type. So I did
create the mapping and place under the config/mapping as explained by the
documentation :
http://www.elastic.co/guide/en/elasticsearch/reference/0.90/mapping-conf-mappings.html
.

For the future, I would like to do the following :

  • Index will be daily : index_name_YYYYMMDD
  • Index will contains N types : index_name_type1, index_name_type2,
    ... index_name_typeN
  • All types will have the same format/mapping.

How can I handle this case with a pre-configured mapping set in the
sub-folder config/mapping ?
I know that I can handle it somehow but putting a default mapping, but
will it work with more than one index case?
Example : I would do a default mapping for all indexes which have a
common prefix (INDEX_A_YYYYMMDD => mapping_A, INDEX_B_YYYYMMDD =>
mapping_B, etc ...)

BR/

--
Please update your bookmarks! We moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a06c0542-1910-4fc7-8a3f-ca899e36fc4d%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/a06c0542-1910-4fc7-8a3f-ca899e36fc4d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to
https://discuss.elastic.co/


You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/182cdd0c-2f15-4f23-aeaa-be184739922a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/182cdd0c-2f15-4f23-aeaa-be184739922a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
Please update your bookmarks! We have moved to https://discuss.elastic.co/

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/563e7704-6066-4b0b-9d09-6771fa03e91a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #6