# Possible? Wildcard template for a collection of fields to solve some dynamic mapping woes
**URL:** https://discuss.elastic.co/t/possible-wildcard-template-for-a-collection-of-fields-to-solve-some-dynamic-mapping-woes/22010
**Category:** Elasticsearch
**Created:** [February 6, 2015, 11:39am UTC](https://discuss.elastic.co/t/possible-wildcard-template-for-a-collection-of-fields-to-solve-some-dynamic-mapping-woes/22010 "2015-02-06T11:39:08Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@Paul\_Kavanagh](https://discuss.elastic.co/u/Paul_Kavanagh)
#### Post date: [February 6, 2015, 11:39am UTC](https://discuss.elastic.co/t/possible-wildcard-template-for-a-collection-of-fields-to-solve-some-dynamic-mapping-woes/22010/1 "2015-02-06T11:39:08Z")
Hi all,
We're having a MapperParsingException problem with some field values when
we get when we use the JSON Filter for Logstash to explode out a JSON
document to Elasticsearch fields.
In 99.9% of cases, certain of these fields are either blank, or contain
dates in the format of yyyy-mm-dd. This allows ES to dynamically map this
field to type dateOptionalTime.
However, we occasionally see non-standard date formats in these fields,
which our main service can handle fine, but which throws a
MapperParsingException in Elasticsearch - such are here:
[2015-02-06 10:46:50,679][WARN][cluster.action.shard] [logging-
production-elasticsearch-ip-xxx-xxx-xxx-148] [logstash-2015.02.06][2]
received shard failed for [logstash-2015.02.06][2], node[
GZpltBjAQUqGyp2B1SLz\_g], [R], s[INITIALIZING], indexUUID [BEdTwj-
QRuOZB713YAQwvA], reason [Failed to start shard, message [
RecoveryFailedException[[logstash-2015.02.06][2]: Recovery failed from [
logging-production-elasticsearch-ip-xxx-xxx-xxx-82][IALW-92RReiLffQjSL3I-g][
logging-production-elasticsearch-ip-xxx-xxx-xxx-82][inet[ip-xxx-xxx-xxx-
82.ec2.internal/xxx.xxx.xxx.82:9300]]{max\_local\_storage\_nodes=1,
aws\_availability\_zone=us-east-1e, aws\_az=us-east-1e} into [logging-
production-elasticsearch-ip-xxx-xxx-xxx-148][GZpltBjAQUqGyp2B1SLz\_g][logging
-production-elasticsearch-ip-xxx-xxx-xxx-148][inet[ip-xxx.xxx.xxx.
148.ec2.internal/xxx.xxx.xxx.148:9300]]{max\_local\_storage\_nodes=1,
aws\_availability\_zone=us-east-1c, aws\_az=us-east-1c}]; nested:
RemoteTransportException[[logging-production-elasticsearch-ip-xxx-xxx-xxx-82
][inet[/xxx.xxx.xxx.82:9300]][internal:index/shard/recovery/start\_recovery
]]; nested: RecoveryEngineException[[logstash-2015.02.06][2] Phase[2]
Execution failed]; nested: RemoteTransportException[[logging-production-
elasticsearch-ip-xxx-xxx-xxx-148][inet[/xxx.xxx.xxx.148:9300]][internal:
index/shard/recovery/translog\_ops]]; nested: MapperParsingException[failed
to parse [apiservice.logstash.@fields.parameters.start\_time]]; nested:
MapperParsingException[failed to parse date field [Feb 5 2015 12:00 AM],
tried both date format [dateOptionalTime], and timestamp number with locale
[]]; nested: IllegalArgumentException[Invalid format: "Feb 5 2015 12:00 AM"
]; ]]
2015-02-06 10:46:53,685][WARN][cluster.action.shard] [logging-
production-elasticsearch-ip-xxx-xxx-xxx-148] [logstash-2015.02.06][2]
received shard failed for [logstash-2015.02.06][2], node[
GZpltBjAQUqGyp2B1SLz\_g], [R], s[INITIALIZING], indexUUID [BEdTwj-
QRuOZB713YAQwvA], reason [master [logging-production-elasticsearch-ip-xxx-
xxx-xxx-148][GZpltBjAQUqGyp2B1SLz\_g][logging-production-elasticsearch-ip-xxx
-xxx-xxx-148][inet[ip-xxx-xxx-xxx-148.ec2.internal/xxx.xxx.xxx.148:9300]]{
max\_local\_storage\_nodes=1, aws\_availability\_zone=us-east-1c, aws\_az=us-east-
1c} marked shard as initializing, but shard is marked as failed, resend
shard failure]
Our planned solution was to create a template for Logstash indices that
will set these fields to string. But as the field above isn't the only
culprit, and more may be added overtime, it makes more sense to create a
template to map all fields under apiservice.logstash.@fields.parameters.\*
to be string. (We never need to query on user entered data, but it's great
to have logged for debugging)
Is it possible to do this with a template? I could not find a way to do
this via the template documentation on the ES site.
Any guidance would be great!
Thanks,
-Paul
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)
#### Post date: [February 6, 2015, 11:41am UTC](https://discuss.elastic.co/t/possible-wildcard-template-for-a-collection-of-fields-to-solve-some-dynamic-mapping-woes/22010/2 "2015-02-06T11:41:33Z")
You mean something like dynamic templates?
[http://code972.com/blog/2015/02/81-elasticsearch-one-tip-a-day-using-dynamic-templates-to-avoid-rigorous-mappings](http://code972.com/blog/2015/02/81-elasticsearch-one-tip-a-day-using-dynamic-templates-to-avoid-rigorous-mappings)
--
Itamar Syn-Hershko
[http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko)
Freelance Developer & Consultant
[Lucene.NET](http://Lucene.NET) committer and PMC member
On Fri, Feb 6, 2015 at 1:39 PM, Paul Kavanagh [pkavanagh@shopkeep.com](mailto:pkavanagh@shopkeep.com)
wrote:
> Hi all,
> We're having a MapperParsingException problem with some field values when
> we get when we use the JSON Filter for Logstash to explode out a JSON
> document to Elasticsearch fields.
>
> In 99.9% of cases, certain of these fields are either blank, or contain
> dates in the format of yyyy-mm-dd. This allows ES to dynamically map this
> field to type dateOptionalTime.
>
> However, we occasionally see non-standard date formats in these fields,
> which our main service can handle fine, but which throws a
> MapperParsingException in Elasticsearch - such are here:
>
> [2015-02-06 10:46:50,679][WARN][cluster.action.shard] [logging-
> production-elasticsearch-ip-xxx-xxx-xxx-148] [logstash-2015.02.06][2]
> received shard failed for [logstash-2015.02.06][2], node[
> GZpltBjAQUqGyp2B1SLz\_g], [R], s[INITIALIZING], indexUUID [BEdTwj-
> QRuOZB713YAQwvA], reason [Failed to start shard, message [
> RecoveryFailedException[[logstash-2015.02.06][2]: Recovery failed from [
> logging-production-elasticsearch-ip-xxx-xxx-xxx-82][IALW-92RReiLffQjSL3I-g
> ][logging-production-elasticsearch-ip-xxx-xxx-xxx-82][inet[ip-xxx-xxx-xxx-
> 82.ec2.internal/xxx.xxx.xxx.82:9300]]{max\_local\_storage\_nodes=1,
> aws\_availability\_zone=us-east-1e, aws\_az=us-east-1e} into [logging-
> production-elasticsearch-ip-xxx-xxx-xxx-148][GZpltBjAQUqGyp2B1SLz\_g][
> logging-production-elasticsearch-ip-xxx-xxx-xxx-148][inet[ip-xxx.xxx.xxx.
> 148.ec2.internal/xxx.xxx.xxx.148:9300]]{max\_local\_storage\_nodes=1,
> aws\_availability\_zone=us-east-1c, aws\_az=us-east-1c}]; nested:
> RemoteTransportException[[logging-production-elasticsearch-ip-xxx-xxx-xxx-
> 82][inet[/xxx.xxx.xxx.82:9300]][internal:index/shard/recovery/
> start\_recovery]]; nested: RecoveryEngineException[[logstash-2015.02.06][2]
> Phase[2] Execution failed]; nested: RemoteTransportException[[logging-
> production-elasticsearch-ip-xxx-xxx-xxx-148][inet[/xxx.xxx.xxx.148:9300]][
> internal:index/shard/recovery/translog\_ops]]; nested:
> MapperParsingException[failed to parse [apiservice.logstash.@fields.
> parameters.start\_time]]; nested: MapperParsingException[failed to parse
> date field [Feb 5 2015 12:00 AM], tried both date format [dateOptionalTime
> ], and timestamp number with locale ]; nested: IllegalArgumentException[
> Invalid format: "Feb 5 2015 12:00 AM"]; ]]
>
> 2015-02-06 10:46:53,685][WARN][cluster.action.shard] [logging-
> production-elasticsearch-ip-xxx-xxx-xxx-148] [logstash-2015.02.06][2]
> received shard failed for [logstash-2015.02.06][2], node[
> GZpltBjAQUqGyp2B1SLz\_g], [R], s[INITIALIZING], indexUUID [BEdTwj-
> QRuOZB713YAQwvA], reason [master [logging-production-elasticsearch-ip-xxx-
> xxx-xxx-148][GZpltBjAQUqGyp2B1SLz\_g][logging-production-elasticsearch-ip-
> xxx-xxx-xxx-148][inet[ip-xxx-xxx-xxx-148.ec2.internal/xxx.xxx.xxx.148:9300
> ]]{max\_local\_storage\_nodes=1, aws\_availability\_zone=us-east-1c, aws\_az=us-
> east-1c} marked shard as initializing, but shard is marked as failed,
> resend shard failure]
>
> Our planned solution was to create a template for Logstash indices that
> will set these fields to string. But as the field above isn't the only
> culprit, and more may be added overtime, it makes more sense to create a
> template to map all fields under apiservice.logstash.@fields.parameters.\*
> to be string. (We never need to query on user entered data, but it's great
> to have logged for debugging)
>
> Is it possible to do this with a template? I could not find a way to do
> this via the template documentation on the ES site.
>
> Any guidance would be great!
>
> Thanks,
> -Paul
>
> --
> You received this message because you are subscribed to the Google Groups
> "elasticsearch" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> To view this discussion on the web visit
> [https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com)
> [https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com?utm_medium=email&utm_source=footer)
> .
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZviZWbhJC83fB-3cm5qmcsuH-ScOo4x-ghS9BZ9t28HCA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZviZWbhJC83fB-3cm5qmcsuH-ScOo4x-ghS9BZ9t28HCA%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@javadevmtl](https://discuss.elastic.co/u/javadevmtl)
#### Post date: [February 6, 2015, 2:52pm UTC](https://discuss.elastic.co/t/possible-wildcard-template-for-a-collection-of-fields-to-solve-some-dynamic-mapping-woes/22010/3 "2015-02-06T14:52:48Z")
A template wont help you here. I mean it's good to use and you should use
them. But once the schema is defined you can't change it. This is no
different then any database.
Your best bet here is to do a bit of data cleansing/normalizing.
If you know that the field is date field and sometimes the date is
different then you have to try to convert it proper date format before
inserting. Especially if you are trying to push it all into one field.
Even if you use wildcards in templates like suggested above, you would have
to know that the date is different to have it pushed to another field.
On Friday, 6 February 2015 06:41:49 UTC-5, Itamar Syn-Hershko wrote:
> You mean something like dynamic templates?
> [Elasticsearch One Tip a Day: Using Dynamic Templates To Avoid Rigorous Mappings](http://code972.com/blog/2015/02/81-elasticsearch-one-tip-a-day-using-dynamic-templates-to-avoid-rigorous-mappings)
>
> --
>
> Itamar Syn-Hershko
> [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko)
> Freelance Developer & Consultant
> [Lucene.NET](http://Lucene.NET) committer and PMC member
>
> On Fri, Feb 6, 2015 at 1:39 PM, Paul Kavanagh \<[pkav...@shopkeep.com](mailto:pkav...@shopkeep.com)
> \\> wrote:
>
> > Hi all,
> > We're having a MapperParsingException problem with some field values when
> > we get when we use the JSON Filter for Logstash to explode out a JSON
> > document to Elasticsearch fields.
> >
> > In 99.9% of cases, certain of these fields are either blank, or contain
> > dates in the format of yyyy-mm-dd. This allows ES to dynamically map this
> > field to type dateOptionalTime.
> >
> > However, we occasionally see non-standard date formats in these fields,
> > which our main service can handle fine, but which throws a
> > MapperParsingException in Elasticsearch - such are here:
> >
> > [2015-02-06 10:46:50,679][WARN][cluster.action.shard] [logging-
> > production-elasticsearch-ip-xxx-xxx-xxx-148] [logstash-2015.02.06][2]
> > received shard failed for [logstash-2015.02.06][2], node[
> > GZpltBjAQUqGyp2B1SLz\_g], [R], s[INITIALIZING], indexUUID [BEdTwj-
> > QRuOZB713YAQwvA], reason [Failed to start shard, message [
> > RecoveryFailedException[[logstash-2015.02.06][2]: Recovery failed from [
> > logging-production-elasticsearch-ip-xxx-xxx-xxx-82][IALW-92RReiLffQjSL3I-
> > g][logging-production-elasticsearch-ip-xxx-xxx-xxx-82][inet[ip-xxx-xxx-
> > xxx-82.ec2.internal/xxx.xxx.xxx.82:9300]]{max\_local\_storage\_nodes=1,
> > aws\_availability\_zone=us-east-1e, aws\_az=us-east-1e} into [logging-
> > production-elasticsearch-ip-xxx-xxx-xxx-148][GZpltBjAQUqGyp2B1SLz\_g][
> > logging-production-elasticsearch-ip-xxx-xxx-xxx-148][inet[ip-xxx.xxx.xxx.
> > 148.ec2.internal/xxx.xxx.xxx.148:9300]]{max\_local\_storage\_nodes=1,
> > aws\_availability\_zone=us-east-1c, aws\_az=us-east-1c}]; nested:
> > RemoteTransportException[[logging-production-elasticsearch-ip-xxx-xxx-xxx
> > -82][inet[/xxx.xxx.xxx.82:9300]][internal:index/shard/recovery/
> > start\_recovery]]; nested: RecoveryEngineException[[logstash-2015.02.06][2
> > ] Phase[2] Execution failed]; nested: RemoteTransportException[[logging-
> > production-elasticsearch-ip-xxx-xxx-xxx-148][inet[/xxx.xxx.xxx.148:9300
> > ]][internal:index/shard/recovery/translog\_ops]]; nested:
> > MapperParsingException[failed to parse [apiservice.logstash.@fields.
> > parameters.start\_time]]; nested: MapperParsingException[failed to parse
> > date field [Feb 5 2015 12:00 AM], tried both date format [
> > dateOptionalTime], and timestamp number with locale ]; nested:
> > IllegalArgumentException[Invalid format: "Feb 5 2015 12:00 AM"]; ]]
> >
> > 2015-02-06 10:46:53,685][WARN][cluster.action.shard] [logging-
> > production-elasticsearch-ip-xxx-xxx-xxx-148] [logstash-2015.02.06][2]
> > received shard failed for [logstash-2015.02.06][2], node[
> > GZpltBjAQUqGyp2B1SLz\_g], [R], s[INITIALIZING], indexUUID [BEdTwj-
> > QRuOZB713YAQwvA], reason [master [logging-production-elasticsearch-ip-xxx
> > -xxx-xxx-148][GZpltBjAQUqGyp2B1SLz\_g][logging-production-elasticsearch-ip
> > -xxx-xxx-xxx-148][inet[ip-xxx-xxx-xxx-148.ec2.internal/xxx.xxx.xxx.148:
> > 9300]]{max\_local\_storage\_nodes=1, aws\_availability\_zone=us-east-1c,
> > aws\_az=us-east-1c} marked shard as initializing, but shard is marked as
> > failed, resend shard failure]
> >
> > Our planned solution was to create a template for Logstash indices that
> > will set these fields to string. But as the field above isn't the only
> > culprit, and more may be added overtime, it makes more sense to create a
> > template to map all fields under apiservice.logstash.@fields.parameters.\*
> > to be string. (We never need to query on user entered data, but it's great
> > to have logged for debugging)
> >
> > Is it possible to do this with a template? I could not find a way to do
> > this via the template documentation on the ES site.
> >
> > Any guidance would be great!
> >
> > Thanks,
> > -Paul
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/130d8b7e-6e8b-43e4-baf8-d17fca8bc7e6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/130d8b7e-6e8b-43e4-baf8-d17fca8bc7e6%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Paul\_Kavanagh](https://discuss.elastic.co/u/Paul_Kavanagh)
#### Post date: [February 6, 2015, 3:01pm UTC](https://discuss.elastic.co/t/possible-wildcard-template-for-a-collection-of-fields-to-solve-some-dynamic-mapping-woes/22010/4 "2015-02-06T15:01:20Z")
I think you have something there. I have come up with this:
curl -XPUT localhost:9200/\_template/template\_1 -d '
{
"template" : "logstash-_",
"order" : 0,
"settings" : {
"number\_of\_shards" : 15
},
"mappings" : {
"dynamic\_templates":[
{"apiservice\_logstash":{
"match":"apiservice.logstash.@fields.parameters._",
"match\_mapping\_type":"dateOptionalTime",
"mapping":{
"type":"string",
"analyzer":"english"
}
}
}
]
}
}
'
However... When I try to post it, Elasticsearch throws:
{"error":"ElasticsearchIllegalArgumentException[Malformed mappings section
for type [dynamic\_templates], should include an inner object describing the
mapping]","status":400}
i've tried a few things, but it doesn't seem to like my mappings block for
some reason.
Any idea why?
On Friday, February 6, 2015 at 11:41:49 AM UTC, Itamar Syn-Hershko wrote:
> You mean something like dynamic templates?
> [Elasticsearch One Tip a Day: Using Dynamic Templates To Avoid Rigorous Mappings](http://code972.com/blog/2015/02/81-elasticsearch-one-tip-a-day-using-dynamic-templates-to-avoid-rigorous-mappings)
>
> --
>
> Itamar Syn-Hershko
> [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko)
> Freelance Developer & Consultant
> [Lucene.NET](http://Lucene.NET) committer and PMC member
>
> On Fri, Feb 6, 2015 at 1:39 PM, Paul Kavanagh \<[pkav...@shopkeep.com](mailto:pkav...@shopkeep.com)
> \\> wrote:
>
> > Hi all,
> > We're having a MapperParsingException problem with some field values when
> > we get when we use the JSON Filter for Logstash to explode out a JSON
> > document to Elasticsearch fields.
> >
> > In 99.9% of cases, certain of these fields are either blank, or contain
> > dates in the format of yyyy-mm-dd. This allows ES to dynamically map this
> > field to type dateOptionalTime.
> >
> > However, we occasionally see non-standard date formats in these fields,
> > which our main service can handle fine, but which throws a
> > MapperParsingException in Elasticsearch - such are here:
> >
> > [2015-02-06 10:46:50,679][WARN][cluster.action.shard] [logging-
> > production-elasticsearch-ip-xxx-xxx-xxx-148] [logstash-2015.02.06][2]
> > received shard failed for [logstash-2015.02.06][2], node[
> > GZpltBjAQUqGyp2B1SLz\_g], [R], s[INITIALIZING], indexUUID [BEdTwj-
> > QRuOZB713YAQwvA], reason [Failed to start shard, message [
> > RecoveryFailedException[[logstash-2015.02.06][2]: Recovery failed from [
> > logging-production-elasticsearch-ip-xxx-xxx-xxx-82][IALW-92RReiLffQjSL3I-
> > g][logging-production-elasticsearch-ip-xxx-xxx-xxx-82][inet[ip-xxx-xxx-
> > xxx-82.ec2.internal/xxx.xxx.xxx.82:9300]]{max\_local\_storage\_nodes=1,
> > aws\_availability\_zone=us-east-1e, aws\_az=us-east-1e} into [logging-
> > production-elasticsearch-ip-xxx-xxx-xxx-148][GZpltBjAQUqGyp2B1SLz\_g][
> > logging-production-elasticsearch-ip-xxx-xxx-xxx-148][inet[ip-xxx.xxx.xxx.
> > 148.ec2.internal/xxx.xxx.xxx.148:9300]]{max\_local\_storage\_nodes=1,
> > aws\_availability\_zone=us-east-1c, aws\_az=us-east-1c}]; nested:
> > RemoteTransportException[[logging-production-elasticsearch-ip-xxx-xxx-xxx
> > -82][inet[/xxx.xxx.xxx.82:9300]][internal:index/shard/recovery/
> > start\_recovery]]; nested: RecoveryEngineException[[logstash-2015.02.06][2
> > ] Phase[2] Execution failed]; nested: RemoteTransportException[[logging-
> > production-elasticsearch-ip-xxx-xxx-xxx-148][inet[/xxx.xxx.xxx.148:9300
> > ]][internal:index/shard/recovery/translog\_ops]]; nested:
> > MapperParsingException[failed to parse [apiservice.logstash.@fields.
> > parameters.start\_time]]; nested: MapperParsingException[failed to parse
> > date field [Feb 5 2015 12:00 AM], tried both date format [
> > dateOptionalTime], and timestamp number with locale ]; nested:
> > IllegalArgumentException[Invalid format: "Feb 5 2015 12:00 AM"]; ]]
> >
> > 2015-02-06 10:46:53,685][WARN][cluster.action.shard] [logging-
> > production-elasticsearch-ip-xxx-xxx-xxx-148] [logstash-2015.02.06][2]
> > received shard failed for [logstash-2015.02.06][2], node[
> > GZpltBjAQUqGyp2B1SLz\_g], [R], s[INITIALIZING], indexUUID [BEdTwj-
> > QRuOZB713YAQwvA], reason [master [logging-production-elasticsearch-ip-xxx
> > -xxx-xxx-148][GZpltBjAQUqGyp2B1SLz\_g][logging-production-elasticsearch-ip
> > -xxx-xxx-xxx-148][inet[ip-xxx-xxx-xxx-148.ec2.internal/xxx.xxx.xxx.148:
> > 9300]]{max\_local\_storage\_nodes=1, aws\_availability\_zone=us-east-1c,
> > aws\_az=us-east-1c} marked shard as initializing, but shard is marked as
> > failed, resend shard failure]
> >
> > Our planned solution was to create a template for Logstash indices that
> > will set these fields to string. But as the field above isn't the only
> > culprit, and more may be added overtime, it makes more sense to create a
> > template to map all fields under apiservice.logstash.@fields.parameters.\*
> > to be string. (We never need to query on user entered data, but it's great
> > to have logged for debugging)
> >
> > Is it possible to do this with a template? I could not find a way to do
> > this via the template documentation on the ES site.
> >
> > Any guidance would be great!
> >
> > Thanks,
> > -Paul
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4932d1e2-9194-431a-8458-afd423866b21%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4932d1e2-9194-431a-8458-afd423866b21%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Paul\_Kavanagh](https://discuss.elastic.co/u/Paul_Kavanagh)
#### Post date: [February 6, 2015, 3:02pm UTC](https://discuss.elastic.co/t/possible-wildcard-template-for-a-collection-of-fields-to-solve-some-dynamic-mapping-woes/22010/5 "2015-02-06T15:02:39Z")
As we create a new index everyday, we're not concerned with retro-applying
the fix to existing indices. So it seems Templates are the way to go here.
On Friday, February 6, 2015 at 2:52:49 PM UTC, John Smith wrote:
> A template wont help you here. I mean it's good to use and you should use
> them. But once the schema is defined you can't change it. This is no
> different then any database.
>
> Your best bet here is to do a bit of data cleansing/normalizing.
>
> If you know that the field is date field and sometimes the date is
> different then you have to try to convert it proper date format before
> inserting. Especially if you are trying to push it all into one field.
>
> Even if you use wildcards in templates like suggested above, you would
> have to know that the date is different to have it pushed to another field.
>
> On Friday, 6 February 2015 06:41:49 UTC-5, Itamar Syn-Hershko wrote:
>
> > You mean something like dynamic templates?
> > [Elasticsearch One Tip a Day: Using Dynamic Templates To Avoid Rigorous Mappings](http://code972.com/blog/2015/02/81-elasticsearch-one-tip-a-day-using-dynamic-templates-to-avoid-rigorous-mappings)
> >
> > --
> >
> > Itamar Syn-Hershko
> > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko)
> > Freelance Developer & Consultant
> > [Lucene.NET](http://Lucene.NET) committer and PMC member
> >
> > On Fri, Feb 6, 2015 at 1:39 PM, Paul Kavanagh [pkav...@shopkeep.com](mailto:pkav...@shopkeep.com)
> > wrote:
> >
> > > Hi all,
> > > We're having a MapperParsingException problem with some field values
> > > when we get when we use the JSON Filter for Logstash to explode out a JSON
> > > document to Elasticsearch fields.
> > >
> > > In 99.9% of cases, certain of these fields are either blank, or contain
> > > dates in the format of yyyy-mm-dd. This allows ES to dynamically map this
> > > field to type dateOptionalTime.
> > >
> > > However, we occasionally see non-standard date formats in these fields,
> > > which our main service can handle fine, but which throws a
> > > MapperParsingException in Elasticsearch - such are here:
> > >
> > > [2015-02-06 10:46:50,679][WARN][cluster.action.shard] [logging-
> > > production-elasticsearch-ip-xxx-xxx-xxx-148] [logstash-2015.02.06][2]
> > > received shard failed for [logstash-2015.02.06][2], node[
> > > GZpltBjAQUqGyp2B1SLz\_g], [R], s[INITIALIZING], indexUUID [BEdTwj-
> > > QRuOZB713YAQwvA], reason [Failed to start shard, message [
> > > RecoveryFailedException[[logstash-2015.02.06][2]: Recovery failed from [
> > > logging-production-elasticsearch-ip-xxx-xxx-xxx-82][IALW-92RReiLffQjSL3I
> > > -g][logging-production-elasticsearch-ip-xxx-xxx-xxx-82][inet[ip-xxx-xxx-
> > > xxx-82.ec2.internal/xxx.xxx.xxx.82:9300]]{max\_local\_storage\_nodes=1,
> > > aws\_availability\_zone=us-east-1e, aws\_az=us-east-1e} into [logging-
> > > production-elasticsearch-ip-xxx-xxx-xxx-148][GZpltBjAQUqGyp2B1SLz\_g][
> > > logging-production-elasticsearch-ip-xxx-xxx-xxx-148][inet[ip-xxx.xxx.xxx
> > > .148.ec2.internal/xxx.xxx.xxx.148:9300]]{max\_local\_storage\_nodes=1,
> > > aws\_availability\_zone=us-east-1c, aws\_az=us-east-1c}]; nested:
> > > RemoteTransportException[[logging-production-elasticsearch-ip-xxx-xxx-
> > > xxx-82][inet[/xxx.xxx.xxx.82:9300]][internal:index/shard/recovery/
> > > start\_recovery]]; nested: RecoveryEngineException[[logstash-2015.02.06][
> > > 2] Phase[2] Execution failed]; nested: RemoteTransportException[[logging
> > > -production-elasticsearch-ip-xxx-xxx-xxx-148][inet[/xxx.xxx.xxx.148:9300
> > > ]][internal:index/shard/recovery/translog\_ops]]; nested:
> > > MapperParsingException[failed to parse [apiservice.logstash.@fields.
> > > parameters.start\_time]]; nested: MapperParsingException[failed to parse
> > > date field [Feb 5 2015 12:00 AM], tried both date format [
> > > dateOptionalTime], and timestamp number with locale ]; nested:
> > > IllegalArgumentException[Invalid format: "Feb 5 2015 12:00 AM"]; ]]
> > >
> > > 2015-02-06 10:46:53,685][WARN][cluster.action.shard] [logging-
> > > production-elasticsearch-ip-xxx-xxx-xxx-148] [logstash-2015.02.06][2]
> > > received shard failed for [logstash-2015.02.06][2], node[
> > > GZpltBjAQUqGyp2B1SLz\_g], [R], s[INITIALIZING], indexUUID [BEdTwj-
> > > QRuOZB713YAQwvA], reason [master [logging-production-elasticsearch-ip-
> > > xxx-xxx-xxx-148][GZpltBjAQUqGyp2B1SLz\_g][logging-production-
> > > elasticsearch-ip-xxx-xxx-xxx-148][inet[ip-xxx-xxx-xxx-148.ec2.internal/
> > > xxx.xxx.xxx.148:9300]]{max\_local\_storage\_nodes=1, aws\_availability\_zone=
> > > us-east-1c, aws\_az=us-east-1c} marked shard as initializing, but shard
> > > is marked as failed, resend shard failure]
> > >
> > > Our planned solution was to create a template for Logstash indices that
> > > will set these fields to string. But as the field above isn't the only
> > > culprit, and more may be added overtime, it makes more sense to create a
> > > template to map all fields under apiservice.logstash.@fields.parameters.\*
> > > to be string. (We never need to query on user entered data, but it's great
> > > to have logged for debugging)
> > >
> > > Is it possible to do this with a template? I could not find a way to do
> > > this via the template documentation on the ES site.
> > >
> > > Any guidance would be great!
> > >
> > > Thanks,
> > > -Paul
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com)
> > > [https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > .
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/da874d3c-a16c-4b46-9aff-ca0f42eee92e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/da874d3c-a16c-4b46-9aff-ca0f42eee92e%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Paul\_Kavanagh](https://discuss.elastic.co/u/Paul_Kavanagh)
#### Post date: [February 9, 2015, 10:23am UTC](https://discuss.elastic.co/t/possible-wildcard-template-for-a-collection-of-fields-to-solve-some-dynamic-mapping-woes/22010/6 "2015-02-09T10:23:49Z")
I think you have something there. I have come up with this:
curl -XPUT localhost:9200/\_template/template\_1 -d '
{
"template" : "logstash-_",
"order" : 0,
"settings" : {
"number\_of\_shards" : 15
},
"mappings" : {
"dynamic\_templates":[
{"apiservice\_logstash":{
"match":"apiservice.logstash.@fields.parameters._",
"match\_mapping\_type":"dateOptionalTime",
"mapping":{
"type":"string",
"analyzer":"english"
}
}
}
]
}
}
'
However... When I try to post it, Elasticsearch throws:
{"error":"ElasticsearchIllegalArgumentException[Malformed mappings section
for type [dynamic\_templates], should include an inner object describing the
mapping]","status":400}
i've tried a few things, but it doesn't seem to like my mappings block for
some reason.
Any idea why?
On Friday, February 6, 2015 at 11:41:49 AM UTC, Itamar Syn-Hershko wrote:
> You mean something like dynamic templates?
> [Elasticsearch One Tip a Day: Using Dynamic Templates To Avoid Rigorous Mappings](http://code972.com/blog/2015/02/81-elasticsearch-one-tip-a-day-using-dynamic-templates-to-avoid-rigorous-mappings)
>
> --
>
> Itamar Syn-Hershko
> [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko)
> Freelance Developer & Consultant
> [Lucene.NET](http://Lucene.NET) committer and PMC member
>
> On Fri, Feb 6, 2015 at 1:39 PM, Paul Kavanagh \<[pkav...@shopkeep.com](mailto:pkav...@shopkeep.com)
> \\> wrote:
>
> > Hi all,
> > We're having a MapperParsingException problem with some field values when
> > we get when we use the JSON Filter for Logstash to explode out a JSON
> > document to Elasticsearch fields.
> >
> > In 99.9% of cases, certain of these fields are either blank, or contain
> > dates in the format of yyyy-mm-dd. This allows ES to dynamically map this
> > field to type dateOptionalTime.
> >
> > However, we occasionally see non-standard date formats in these fields,
> > which our main service can handle fine, but which throws a
> > MapperParsingException in Elasticsearch - such are here:
> >
> > [2015-02-06 10:46:50,679][WARN][cluster.action.shard] [logging-
> > production-elasticsearch-ip-xxx-xxx-xxx-148] [logstash-2015.02.06][2]
> > received shard failed for [logstash-2015.02.06][2], node[
> > GZpltBjAQUqGyp2B1SLz\_g], [R], s[INITIALIZING], indexUUID [BEdTwj-
> > QRuOZB713YAQwvA], reason [Failed to start shard, message [
> > RecoveryFailedException[[logstash-2015.02.06][2]: Recovery failed from [
> > logging-production-elasticsearch-ip-xxx-xxx-xxx-82][IALW-92RReiLffQjSL3I-
> > g][logging-production-elasticsearch-ip-xxx-xxx-xxx-82][inet[ip-xxx-xxx-
> > xxx-82.ec2.internal/xxx.xxx.xxx.82:9300]]{max\_local\_storage\_nodes=1,
> > aws\_availability\_zone=us-east-1e, aws\_az=us-east-1e} into [logging-
> > production-elasticsearch-ip-xxx-xxx-xxx-148][GZpltBjAQUqGyp2B1SLz\_g][
> > logging-production-elasticsearch-ip-xxx-xxx-xxx-148][inet[ip-xxx.xxx.xxx.
> > 148.ec2.internal/xxx.xxx.xxx.148:9300]]{max\_local\_storage\_nodes=1,
> > aws\_availability\_zone=us-east-1c, aws\_az=us-east-1c}]; nested:
> > RemoteTransportException[[logging-production-elasticsearch-ip-xxx-xxx-xxx
> > -82][inet[/xxx.xxx.xxx.82:9300]][internal:index/shard/recovery/
> > start\_recovery]]; nested: RecoveryEngineException[[logstash-2015.02.06][2
> > ] Phase[2] Execution failed]; nested: RemoteTransportException[[logging-
> > production-elasticsearch-ip-xxx-xxx-xxx-148][inet[/xxx.xxx.xxx.148:9300
> > ]][internal:index/shard/recovery/translog\_ops]]; nested:
> > MapperParsingException[failed to parse [apiservice.logstash.@fields.
> > parameters.start\_time]]; nested: MapperParsingException[failed to parse
> > date field [Feb 5 2015 12:00 AM], tried both date format [
> > dateOptionalTime], and timestamp number with locale ]; nested:
> > IllegalArgumentException[Invalid format: "Feb 5 2015 12:00 AM"]; ]]
> >
> > 2015-02-06 10:46:53,685][WARN][cluster.action.shard] [logging-
> > production-elasticsearch-ip-xxx-xxx-xxx-148] [logstash-2015.02.06][2]
> > received shard failed for [logstash-2015.02.06][2], node[
> > GZpltBjAQUqGyp2B1SLz\_g], [R], s[INITIALIZING], indexUUID [BEdTwj-
> > QRuOZB713YAQwvA], reason [master [logging-production-elasticsearch-ip-xxx
> > -xxx-xxx-148][GZpltBjAQUqGyp2B1SLz\_g][logging-production-elasticsearch-ip
> > -xxx-xxx-xxx-148][inet[ip-xxx-xxx-xxx-148.ec2.internal/xxx.xxx.xxx.148:
> > 9300]]{max\_local\_storage\_nodes=1, aws\_availability\_zone=us-east-1c,
> > aws\_az=us-east-1c} marked shard as initializing, but shard is marked as
> > failed, resend shard failure]
> >
> > Our planned solution was to create a template for Logstash indices that
> > will set these fields to string. But as the field above isn't the only
> > culprit, and more may be added overtime, it makes more sense to create a
> > template to map all fields under apiservice.logstash.@fields.parameters.\*
> > to be string. (We never need to query on user entered data, but it's great
> > to have logged for debugging)
> >
> > Is it possible to do this with a template? I could not find a way to do
> > this via the template documentation on the ES site.
> >
> > Any guidance would be great!
> >
> > Thanks,
> > -Paul
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \.
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6ab11e08-873b-4610-9573-d8781327aeb2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6ab11e08-873b-4610-9573-d8781327aeb2%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)
#### Post date: [February 9, 2015, 10:24am UTC](https://discuss.elastic.co/t/possible-wildcard-template-for-a-collection-of-fields-to-solve-some-dynamic-mapping-woes/22010/7 "2015-02-09T10:24:37Z")
yes, you are using string properties on a date mapping field
--
Itamar Syn-Hershko
[http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko)
Freelance Developer & Consultant
[Lucene.NET](http://Lucene.NET) committer and PMC member
On Mon, Feb 9, 2015 at 12:23 PM, Paul Kavanagh [pkavanagh@shopkeep.com](mailto:pkavanagh@shopkeep.com)
wrote:
> I think you have something there. I have come up with this:
>
> curl -XPUT localhost:9200/\_template/template\_1 -d '
> {
> "template" : "logstash-_",
> "order" : 0,
> "settings" : {
> "number\_of\_shards" : 15
> },
> "mappings" : {
> "dynamic\_templates":[
> {"apiservice\_logstash":{
> "match":"apiservice.logstash.@fields.parameters._",
> "match\_mapping\_type":"dateOptionalTime",
> "mapping":{
> "type":"string",
> "analyzer":"english"
> }
> }
> }
> ]
> }
> }
> '
>
> However... When I try to post it, Elasticsearch throws:
> {"error":"ElasticsearchIllegalArgumentException[Malformed mappings section
> for type [dynamic\_templates], should include an inner object describing the
> mapping]","status":400}
>
> i've tried a few things, but it doesn't seem to like my mappings block for
> some reason.
>
> Any idea why?
>
> On Friday, February 6, 2015 at 11:41:49 AM UTC, Itamar Syn-Hershko wrote:
>
> > You mean something like dynamic templates? [http://code972.com/](http://code972.com/)
> > blog/2015/02/81-elasticsearch-one-tip-a-day-using-dynamic-
> > templates-to-avoid-rigorous-mappings
> >
> > --
> >
> > Itamar Syn-Hershko
> > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko)
> > Freelance Developer & Consultant
> > [Lucene.NET](http://Lucene.NET) committer and PMC member
> >
> > On Fri, Feb 6, 2015 at 1:39 PM, Paul Kavanagh [pkav...@shopkeep.com](mailto:pkav...@shopkeep.com)
> > wrote:
> >
> > > Hi all,
> > > We're having a MapperParsingException problem with some field values
> > > when we get when we use the JSON Filter for Logstash to explode out a JSON
> > > document to Elasticsearch fields.
> > >
> > > In 99.9% of cases, certain of these fields are either blank, or contain
> > > dates in the format of yyyy-mm-dd. This allows ES to dynamically map this
> > > field to type dateOptionalTime.
> > >
> > > However, we occasionally see non-standard date formats in these fields,
> > > which our main service can handle fine, but which throws a
> > > MapperParsingException in Elasticsearch - such are here:
> > >
> > > [2015-02-06 10:46:50,679][WARN][cluster.action.shard] [logging-
> > > production-elasticsearch-ip-xxx-xxx-xxx-148] [logstash-2015.02.06][2]
> > > received shard failed for [logstash-2015.02.06][2], node[
> > > GZpltBjAQUqGyp2B1SLz\_g], [R], s[INITIALIZING], indexUUID [BEdTwj-
> > > QRuOZB713YAQwvA], reason [Failed to start shard, message [
> > > RecoveryFailedException[[logstash-2015.02.06][2]: Recovery failed from [
> > > logging-production-elasticsearch-ip-xxx-xxx-xxx-82][IALW-92RReiLffQjSL3I
> > > -g][logging-production-elasticsearch-ip-xxx-xxx-xxx-82][inet[ip-xxx-xxx-
> > > xxx-82.ec2.internal/xxx.xxx.xxx.82:9300]]{max\_local\_storage\_nodes=1,
> > > aws\_availability\_zone=us-east-1e, aws\_az=us-east-1e} into [logging-
> > > production-elasticsearch-ip-xxx-xxx-xxx-148][GZpltBjAQUqGyp2B1SLz\_g][
> > > logging-production-elasticsearch-ip-xxx-xxx-xxx-148][inet[ip-xxx.xxx.xxx
> > > .148.ec2.internal/xxx.xxx.xxx.148:9300]]{max\_local\_storage\_nodes=1,
> > > aws\_availability\_zone=us-east-1c, aws\_az=us-east-1c}]; nested:
> > > RemoteTransportException[[logging-production-elasticsearch-ip-xxx-xxx-
> > > xxx-82][inet[/xxx.xxx.xxx.82:9300]][internal:index/shard/recovery/
> > > start\_recovery]]; nested: RecoveryEngineException[[logstash-2015.02.06][
> > > 2] Phase[2] Execution failed]; nested: RemoteTransportException[[logging
> > > -production-elasticsearch-ip-xxx-xxx-xxx-148][inet[/xxx.xxx.xxx.148:9300
> > > ]][internal:index/shard/recovery/translog\_ops]]; nested:
> > > MapperParsingException[failed to parse [apiservice.logstash.@fields.p
> > > arameters.start\_time]]; nested: MapperParsingException[failed to parse
> > > date field [Feb 5 2015 12:00 AM], tried both date format [
> > > dateOptionalTime], and timestamp number with locale ]; nested:
> > > IllegalArgumentException[Invalid format: "Feb 5 2015 12:00 AM"]; ]]
> > >
> > > 2015-02-06 10:46:53,685][WARN][cluster.action.shard] [logging-
> > > production-elasticsearch-ip-xxx-xxx-xxx-148] [logstash-2015.02.06][2]
> > > received shard failed for [logstash-2015.02.06][2], node[
> > > GZpltBjAQUqGyp2B1SLz\_g], [R], s[INITIALIZING], indexUUID [BEdTwj-
> > > QRuOZB713YAQwvA], reason [master [logging-production-elasticsearch-ip-
> > > xxx-xxx-xxx-148][GZpltBjAQUqGyp2B1SLz\_g][logging-production-
> > > elasticsearch-ip-xxx-xxx-xxx-148][inet[ip-xxx-xxx-xxx-148.ec2.internal/
> > > xxx.xxx.xxx.148:9300]]{max\_local\_storage\_nodes=1, aws\_availability\_zone=
> > > us-east-1c, aws\_az=us-east-1c} marked shard as initializing, but shard
> > > is marked as failed, resend shard failure]
> > >
> > > Our planned solution was to create a template for Logstash indices that
> > > will set these fields to string. But as the field above isn't the only
> > > culprit, and more may be added overtime, it makes more sense to create a
> > > template to map all fields under apiservice.logstash.@fields.parameters.\*
> > > to be string. (We never need to query on user entered data, but it's great
> > > to have logged for debugging)
> > >
> > > Is it possible to do this with a template? I could not find a way to do
> > > this via the template documentation on the ES site.
> > >
> > > Any guidance would be great!
> > >
> > > Thanks,
> > > -Paul
> > >
> > > --
> > > You received this message because you are subscribed to the Google
> > > Groups "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send
> > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%
> > > [40googlegroups.com](http://40googlegroups.com)
> > > [https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > .
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "elasticsearch" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > To view this discussion on the web visit
> > [https://groups.google.com/d/msgid/elasticsearch/6ab11e08-873b-4610-9573-d8781327aeb2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6ab11e08-873b-4610-9573-d8781327aeb2%40googlegroups.com)
> > [https://groups.google.com/d/msgid/elasticsearch/6ab11e08-873b-4610-9573-d8781327aeb2%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6ab11e08-873b-4610-9573-d8781327aeb2%40googlegroups.com?utm_medium=email&utm_source=footer)
> > .
>
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsrLZV2JMEAdyQx5GOzbac5QmzG%2Bni-CJXP8Brky4aYtA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsrLZV2JMEAdyQx5GOzbac5QmzG%2Bni-CJXP8Brky4aYtA%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)
#### Post date: [February 9, 2015, 10:24am UTC](https://discuss.elastic.co/t/possible-wildcard-template-for-a-collection-of-fields-to-solve-some-dynamic-mapping-woes/22010/8 "2015-02-09T10:24:55Z")
Please refer to
> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.
--
Itamar Syn-Hershko
[http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko)
Freelance Developer & Consultant
[Lucene.NET](http://Lucene.NET) committer and PMC member
On Mon, Feb 9, 2015 at 12:24 PM, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com)
wrote:
> yes, you are using string properties on a date mapping field
>
> --
>
> Itamar Syn-Hershko
> [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko)
> Freelance Developer & Consultant
> [Lucene.NET](http://Lucene.NET) committer and PMC member
>
> On Mon, Feb 9, 2015 at 12:23 PM, Paul Kavanagh [pkavanagh@shopkeep.com](mailto:pkavanagh@shopkeep.com)
> wrote:
>
> > I think you have something there. I have come up with this:
> >
> > curl -XPUT localhost:9200/\_template/template\_1 -d '
> > {
> > "template" : "logstash-_",
> > "order" : 0,
> > "settings" : {
> > "number\_of\_shards" : 15
> > },
> > "mappings" : {
> > "dynamic\_templates":[
> > {"apiservice\_logstash":{
> > "match":"apiservice.logstash.@fields.parameters._",
> > "match\_mapping\_type":"dateOptionalTime",
> > "mapping":{
> > "type":"string",
> > "analyzer":"english"
> > }
> > }
> > }
> > ]
> > }
> > }
> > '
> >
> > However... When I try to post it, Elasticsearch throws:
> > {"error":"ElasticsearchIllegalArgumentException[Malformed mappings
> > section for type [dynamic\_templates], should include an inner object
> > describing the mapping]","status":400}
> >
> > i've tried a few things, but it doesn't seem to like my mappings block
> > for some reason.
> >
> > Any idea why?
> >
> > On Friday, February 6, 2015 at 11:41:49 AM UTC, Itamar Syn-Hershko wrote:
> >
> > > You mean something like dynamic templates? [http://code972.com/](http://code972.com/)
> > > blog/2015/02/81-elasticsearch-one-tip-a-day-using-dynamic-
> > > templates-to-avoid-rigorous-mappings
> > >
> > > --
> > >
> > > Itamar Syn-Hershko
> > > [http://code972.com](http://code972.com) | @synhershko [https://twitter.com/synhershko](https://twitter.com/synhershko)
> > > Freelance Developer & Consultant
> > > [Lucene.NET](http://Lucene.NET) committer and PMC member
> > >
> > > On Fri, Feb 6, 2015 at 1:39 PM, Paul Kavanagh [pkav...@shopkeep.com](mailto:pkav...@shopkeep.com)
> > > wrote:
> > >
> > > > Hi all,
> > > > We're having a MapperParsingException problem with some field values
> > > > when we get when we use the JSON Filter for Logstash to explode out a JSON
> > > > document to Elasticsearch fields.
> > > >
> > > > In 99.9% of cases, certain of these fields are either blank, or contain
> > > > dates in the format of yyyy-mm-dd. This allows ES to dynamically map this
> > > > field to type dateOptionalTime.
> > > >
> > > > However, we occasionally see non-standard date formats in these fields,
> > > > which our main service can handle fine, but which throws a
> > > > MapperParsingException in Elasticsearch - such are here:
> > > >
> > > > [2015-02-06 10:46:50,679][WARN][cluster.action.shard] [logging-
> > > > production-elasticsearch-ip-xxx-xxx-xxx-148] [logstash-2015.02.06][2]
> > > > received shard failed for [logstash-2015.02.06][2], node[
> > > > GZpltBjAQUqGyp2B1SLz\_g], [R], s[INITIALIZING], indexUUID [BEdTwj-
> > > > QRuOZB713YAQwvA], reason [Failed to start shard, message [
> > > > RecoveryFailedException[[logstash-2015.02.06][2]: Recovery failed from
> > > > [logging-production-elasticsearch-ip-xxx-xxx-xxx-82][IALW-92
> > > > RReiLffQjSL3I-g][logging-production-elasticsearch-ip-xxx-xxx-xxx-82][
> > > > inet[ip-xxx-xxx-xxx-82.ec2.internal/xxx.xxx.xxx.82:9300]]{
> > > > max\_local\_storage\_nodes=1, aws\_availability\_zone=us-east-1e, aws\_az=us-
> > > > east-1e} into [logging-production-elasticsearch-ip-xxx-xxx-xxx-148][
> > > > GZpltBjAQUqGyp2B1SLz\_g][logging-production-elasticsearch-ip-xxx-xxx-xxx
> > > > -148][inet[ip-xxx.xxx.xxx.148.ec2.internal/xxx.xxx.xxx.148:9300]]{
> > > > max\_local\_storage\_nodes=1, aws\_availability\_zone=us-east-1c, aws\_az=us-
> > > > east-1c}]; nested: RemoteTransportException[[logging-production-
> > > > elasticsearch-ip-xxx-xxx-xxx-82][inet[/xxx.xxx.xxx.82:9300]][internal:
> > > > index/shard/recovery/start\_recovery]]; nested: RecoveryEngineException
> > > > [[logstash-2015.02.06][2] Phase[2] Execution failed]; nested:
> > > > RemoteTransportException[[logging-production-elasticsearch-ip-xxx-xxx-
> > > > xxx-148][inet[/xxx.xxx.xxx.148:9300]][internal:index/shard/recovery/
> > > > translog\_ops]]; nested: MapperParsingException[failed to parse [
> > > > apiservice.logstash.@fields.parameters.start\_time]]; nested:
> > > > MapperParsingException[failed to parse date field [Feb 5 2015 12:00 AM
> > > > ], tried both date format [dateOptionalTime], and timestamp number with
> > > > locale ]; nested: IllegalArgumentException[Invalid format: "Feb 5
> > > > 2015 12:00 AM"]; ]]
> > > >
> > > > 2015-02-06 10:46:53,685][WARN][cluster.action.shard] [logging-
> > > > production-elasticsearch-ip-xxx-xxx-xxx-148] [logstash-2015.02.06][2]
> > > > received shard failed for [logstash-2015.02.06][2], node[
> > > > GZpltBjAQUqGyp2B1SLz\_g], [R], s[INITIALIZING], indexUUID [BEdTwj-
> > > > QRuOZB713YAQwvA], reason [master [logging-production-elasticsearch-ip-
> > > > xxx-xxx-xxx-148][GZpltBjAQUqGyp2B1SLz\_g][logging-production-
> > > > elasticsearch-ip-xxx-xxx-xxx-148][inet[ip-xxx-xxx-xxx-148.ec2.internal/
> > > > xxx.xxx.xxx.148:9300]]{max\_local\_storage\_nodes=1, aws\_availability\_zone
> > > > =us-east-1c, aws\_az=us-east-1c} marked shard as initializing, but
> > > > shard is marked as failed, resend shard failure]
> > > >
> > > > Our planned solution was to create a template for Logstash indices that
> > > > will set these fields to string. But as the field above isn't the only
> > > > culprit, and more may be added overtime, it makes more sense to create a
> > > > template to map all fields under apiservice.logstash.@fields.parameters.\*
> > > > to be string. (We never need to query on user entered data, but it's great
> > > > to have logged for debugging)
> > > >
> > > > Is it possible to do this with a template? I could not find a way to do
> > > > this via the template documentation on the ES site.
> > > >
> > > > Any guidance would be great!
> > > >
> > > > Thanks,
> > > > -Paul
> > > >
> > > > --
> > > > You received this message because you are subscribed to the Google
> > > > Groups "elasticsearch" group.
> > > > To unsubscribe from this group and stop receiving emails from it, send
> > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com).
> > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/)
> > > > msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%
> > > > [40googlegroups.com](http://40googlegroups.com)
> > > > [https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6ca4030f-b6bb-4907-b2fc-e3166fa2a6af%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > > .
> > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
> > >
> > > --
> > > You received this message because you are subscribed to the Google Groups
> > > "elasticsearch" group.
> > > To unsubscribe from this group and stop receiving emails from it, send an
> > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
> > > To view this discussion on the web visit
> > > [https://groups.google.com/d/msgid/elasticsearch/6ab11e08-873b-4610-9573-d8781327aeb2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6ab11e08-873b-4610-9573-d8781327aeb2%40googlegroups.com)
> > > [https://groups.google.com/d/msgid/elasticsearch/6ab11e08-873b-4610-9573-d8781327aeb2%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6ab11e08-873b-4610-9573-d8781327aeb2%40googlegroups.com?utm_medium=email&utm_source=footer)
> > > .
> >
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsLpbjxRF1WA3Zhub76V%2BCOrj5LRB4Xb4yKDY%2Bj%3DEGawQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsLpbjxRF1WA3Zhub76V%2BCOrj5LRB4Xb4yKDY%2Bj%3DEGawQ%40mail.gmail.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 12:34am UTC](https://discuss.elastic.co/t/possible-wildcard-template-for-a-collection-of-fields-to-solve-some-dynamic-mapping-woes/22010/9 "2017-07-06T00:34:14Z")