Index template requires settings object even if its value is empty


(Brian Yoder) #1

I am not sure if this is a problem or if it's OK.

Working with the ELK stack I have switched direction, and instead of
locking down the Elasticsearch mappings I am now using its automatic
mapping functions. And by adding the following JSON template definition to
the /path.to.config/templates/automap.json file I can get numeric fields
automatically correctly mapped even though logstash always emits their
values as strings ("45.6" instead of 45.6). Very nice!

{
"automap" : {
"template" : "*",
"settings" : { },
"mappings" : {
"default" : {
"numeric_detection" : true,
"properties" : {
"message" : {"type" : "string"},
"host" : {"type" : "string"},
"@version" : {"type" : "string"}
}
}
}
}
}

When I removed the "settings":{} entirely, it was as if the template did
not exist; the numeric detection was not enabled and all string values were
seen as strings even if they contained numbers. Because all of the settings
are being controlled within elasticsearch.yml and not the template (e.g.
number of shards, number of replicas, and so on), eliminating the settings
from the template is desired, even if I have to leave it in but set its
value to the empty JSON object.

If this is the way it's supposed to work, that's OK. But I couldn't find
anything in the documentation about it, and just wanted to get a
verification either way.

Thanks!

Brian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ff4afb8e-c3e4-4772-aa48-bd6a651c78e8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Alexander Reelsen) #2

Hey,

which ES version are you using? Seems to work with the latest version. You
can also use the index template API, so you do not have to fiddle with
local files (and copy them when adding new nodes).

PUT _template/automap
{
"template": "*",
"mappings": {
"default": {
"numeric_detection": true,
"properties": {
"message": {
"type": "string"
},
"host": {
"type": "string"
},
"@version": {
"type": "string"
}
}
}
}
}

--Alex

On Tue, Jun 3, 2014 at 5:57 PM, Brian brian.from.fl@gmail.com wrote:

I am not sure if this is a problem or if it's OK.

Working with the ELK stack I have switched direction, and instead of
locking down the Elasticsearch mappings I am now using its automatic
mapping functions. And by adding the following JSON template definition to
the /path.to.config/templates/automap.json file I can get numeric
fields automatically correctly mapped even though logstash always emits
their values as strings ("45.6" instead of 45.6). Very nice!

{
"automap" : {
"template" : "*",
"settings" : { },
"mappings" : {
"default" : {
"numeric_detection" : true,
"properties" : {
"message" : {"type" : "string"},
"host" : {"type" : "string"},
"@version" : {"type" : "string"}
}
}
}
}
}

When I removed the "settings":{} entirely, it was as if the template
did not exist; the numeric detection was not enabled and all string values
were seen as strings even if they contained numbers. Because all of the
settings are being controlled within elasticsearch.yml and not the template
(e.g. number of shards, number of replicas, and so on), eliminating the
settings from the template is desired, even if I have to leave it in but
set its value to the empty JSON object.

If this is the way it's supposed to work, that's OK. But I couldn't find
anything in the documentation about it, and just wanted to get a
verification either way.

Thanks!

Brian

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ff4afb8e-c3e4-4772-aa48-bd6a651c78e8%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/ff4afb8e-c3e4-4772-aa48-bd6a651c78e8%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM9g43jE8cKehZLNxBNbFZs2jrpw%3DRnbC5b%3DO10uJSr5Pg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Brian Yoder) #3

Alex,

I am running ES version 1.2.1.

It seemed to work (no errors in the logs), but I did it as an on-disk
template and not via PUT. And without the settings, it behaved as if it
wasn't there.

The question is now moot, because I actually need the following setting:

"settings" : {
"index.mapping.ignore_malformed" : true,
"index.query.default_field" : "message"
},

I don't have a problem fiddling with local files; Elasticsearch, the
wrapper script, and everything else I need is stored in a single zip
archive that our operations team can easily install. So once I install it
on my laptop and verify that it's working, it's 100.0000% repeatable when
installed on any QA or production server.

I also configure logstash's elasticsearch_http as follows:

manage_template => false

That way, I don't have to depend on logstash (or anything else) doing that
for me. It's already done by the base ES install package.

Brian

On Monday, June 16, 2014 8:03:33 AM UTC-4, Alexander Reelsen wrote:

Hey,

which ES version are you using? Seems to work with the latest version. You
can also use the index template API, so you do not have to fiddle with
local files (and copy them when adding new nodes).

PUT _template/automap
{
"template": "*",
"mappings": {
"default": {
"numeric_detection": true,
"properties": {
"message": {
"type": "string"
},
"host": {
"type": "string"
},
"@version": {
"type": "string"
}
}
}
}
}

--Alex

On Tue, Jun 3, 2014 at 5:57 PM, Brian <brian....@gmail.com <javascript:>>
wrote:

I am not sure if this is a problem or if it's OK.

Working with the ELK stack I have switched direction, and instead of
locking down the Elasticsearch mappings I am now using its automatic
mapping functions. And by adding the following JSON template definition to
the /path.to.config/templates/automap.json file I can get numeric
fields automatically correctly mapped even though logstash always emits
their values as strings ("45.6" instead of 45.6). Very nice!

{
"automap" : {
"template" : "*",
"settings" : { },
"mappings" : {
"default" : {
"numeric_detection" : true,
"properties" : {
"message" : {"type" : "string"},
"host" : {"type" : "string"},
"@version" : {"type" : "string"}
}
}
}
}
}

When I removed the "settings":{} entirely, it was as if the template
did not exist; the numeric detection was not enabled and all string values
were seen as strings even if they contained numbers. Because all of the
settings are being controlled within elasticsearch.yml and not the template
(e.g. number of shards, number of replicas, and so on), eliminating the
settings from the template is desired, even if I have to leave it in but
set its value to the empty JSON object.

If this is the way it's supposed to work, that's OK. But I couldn't find
anything in the documentation about it, and just wanted to get a
verification either way.

Thanks!

Brian

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ff4afb8e-c3e4-4772-aa48-bd6a651c78e8%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/ff4afb8e-c3e4-4772-aa48-bd6a651c78e8%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0ffa60d5-92a1-462f-b335-de83907060eb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Brian Yoder) #4

By the way, I got a little ahead of myself in the previous post. In
particular:

"settings" : {
"index.mapping.ignore_malformed" : true*,*
"index.query.default_field" : "message"
},

Apparently, when added the setting above in red, and then removed the
following option from my ES 1.2.1 start-up script, Kibana was no longer
able to search on HTTP and it required message:HTTP because the _all field
has also been disabled:

-Des.index.query.default_field=message

So I put the configuration option (above) back into my ES start-up script,
and removed the index configuration option in red above (as it didn't seem
to work). Not sure if this is a problem with my understanding (most likely)
or a bug in ES (very unlikely). But I offer it to the experts for comment
and correction.

But however it should be, ES rocks and I've managed to get several people
up and running with a one-button (as it were) build, install, load, and
test. Awesome job, Elasticsearch.com! You make me look good!

Brian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d68e3db5-e651-4e57-85b8-fea70a5e8de9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #5