Force default "type" of new fields to be "string"?

Hi, I'm currently setting the type of the "known" fields in my index via
elasticsearch-0.19.3/config/templates/foo.json:

{
"foo_template" : {
"template" : "*",
"settings" : {
"index.number_of_shards" : 3,
"index.number_of_replicas" : 1
},
"mappings" : {
"default" : {
"_source" : { "enabled" : true,
"compress" : true },
"_all" : { "enabled" : false },
"_timestamp" : { "enabled" : true,
"store" : true },
"analyzer" : "keyword",
"properties" : {
"name" : { "type" : "string", "analyzer" : "keyword" },
"size" : { "type" : "long" }
}
}
}
}
}

Problem is that I also index user-supplied fields, but don't know the names
of the field beforehand. E.g.:

$ curl -XPOST 'http://localhost:9200/test/123' -d '{ "name" : "test", "x-foo-meta"
: "2013-01-01"
, "size" : 1 }'

But I don't want "x-foo-meta" type to ever be date, only string. If the
first insert happens to contain a value that looks like a date then the
mapping for the field is permanently set to date:

$ curl -XGET 'http://localhost:9200/test/_mapping'
{"test":{"123":{"analyzer":"keyword","_all":{"enabled":false},"_timestamp":{"store":"yes","enabled":true},"_source":{"compress":true},"properties":{"name":{"type":"string","analyzer":"keyword"},"size":{"type":"long"},
"x-foo-meta":{"type":"date"
,"format":"dateOptionalTime"}}},"default":{"analyzer":"keyword","_all":{"enabled":false},"_timestamp":{"store":"yes","enabled":true},"_source":{"compress":true},"properties":{"name":{"type":"string","analyzer":"keyword"},"size":{"type":"long"}}}}}

If I try to insert a non-date value I get an error:

$ curl -XPOST 'http://localhost:9200/test/123' -d '{ "name" : "test", "x-foo-meta"
: "x2013-01-01"
, "size" : 1 }'
{"error":"MapperParsingException[Failed to parse [x-foo-meta]]; nested:
MapperParsingException[failed to parse date field [x2013-01-01], tried both
date format [dateOptionalTime], and timestamp number]; nested:
IllegalArgumentException[Invalid format: "x2013-01-01"]; ","status":400}

So, how can I force the type of all unspecified fields to be "string"?
Wildcards don't work in "properties", and setting "type" : "string" in the
foo.json caused an error on insert, I assume because the "size" field I'm
inserting is not a string:

{"error":"MapperParsingException[mapping [default]]; nested:
MapperParsingException[Trying to parse an object but has a different type
[string] for [default]]; ","status":400}

Thanks,
Jamshid

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Dynamic templates is what you are looking for:

Looking for dynamic_templates toward the bottom. That section really should
be a standalone page IMHO.

--
Ivan

On Wed, Apr 17, 2013 at 11:47 AM, Jamshid jamshid69@gmail.com wrote:

Hi, I'm currently setting the type of the "known" fields in my index via
elasticsearch-0.19.3/config/templates/foo.json:

{
"foo_template" : {
"template" : "*",
"settings" : {
"index.number_of_shards" : 3,
"index.number_of_replicas" : 1
},
"mappings" : {
"default" : {
"_source" : { "enabled" : true,
"compress" : true },
"_all" : { "enabled" : false },
"_timestamp" : { "enabled" : true,
"store" : true },
"analyzer" : "keyword",
"properties" : {
"name" : { "type" : "string", "analyzer" : "keyword"
},
"size" : { "type" : "long" }
}
}
}
}
}

Problem is that I also index user-supplied fields, but don't know the
names of the field beforehand. E.g.:

$ curl -XPOST 'http://localhost:9200/test/123' -d '{ "name" : "test", "x-foo-meta"
: "2013-01-01"
, "size" : 1 }'

But I don't want "x-foo-meta" type to ever be date, only string. If the
first insert happens to contain a value that looks like a date then the
mapping for the field is permanently set to date:

$ curl -XGET 'http://localhost:9200/test/_mapping'

{"test":{"123":{"analyzer":"keyword","_all":{"enabled":false},"_timestamp":{"store":"yes","enabled":true},"_source":{"compress":true},"properties":{"name":{"type":"string","analyzer":"keyword"},"size":{"type":"long"},
"x-foo-meta":{"type":"date"
,"format":"dateOptionalTime"}}},"default":{"analyzer":"keyword","_all":{"enabled":false},"_timestamp":{"store":"yes","enabled":true},"_source":{"compress":true},"properties":{"name":{"type":"string","analyzer":"keyword"},"size":{"type":"long"}}}}}

If I try to insert a non-date value I get an error:

$ curl -XPOST 'http://localhost:9200/test/123' -d '{ "name" : "test", "x-foo-meta"
: "x2013-01-01"
, "size" : 1 }'
{"error":"MapperParsingException[Failed to parse [x-foo-meta]]; nested:
MapperParsingException[failed to parse date field [x2013-01-01], tried both
date format [dateOptionalTime], and timestamp number]; nested:
IllegalArgumentException[Invalid format: "x2013-01-01"]; ","status":400}

So, how can I force the type of all unspecified fields to be "string"?
Wildcards don't work in "properties", and setting "type" : "string" in the
foo.json caused an error on insert, I assume because the "size" field I'm
inserting is not a string:

{"error":"MapperParsingException[mapping [default]]; nested:
MapperParsingException[Trying to parse an object but has a different type
[string] for [default]]; ","status":400}

Thanks,
Jamshid

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Thanks, Ivan, yes that is exactly what I need. Had trouble figuring out
where to merge "dynamic_templates" into my existing
elasticsearch-0.19.3/config/templates/foo.json, but finally got it:

{
"foo_template" : {
"template" : "",
"settings" : {
"index.number_of_shards" : 3,
"index.number_of_replicas" : 1
},
"mappings" : {
"default" : {
"_source" : { "enabled" : true,
"compress" : true },
"_all" : { "enabled" : false },
"_timestamp" : { "enabled" : true,
"store" : true },
"analyzer" : "keyword",
"dynamic_templates" : [ {
"thisnamedoesnotmatter": {
"match" : "x
",
"mapping" : {
"type": "string",
"analyzer": "keyword"
}
}
} ],
"properties" : {
"name" : { "type" : "string", "analyzer" : "keyword" },
"size" : { "type" : "long" }
}
}
}
}
}

Now size remains a "long" but other fields are forced to "string",
regardless of the type of of the first value submitted.

curl -XDELETE 'http://localhost:9200/test'
curl -XPOST 'http://localhst:9200/test/123' -d '{ "name" : "test",
"x-foo-meta" : "2013-01-01", "size" : "1" }'
curl -XGET 'http://localhost:9200/test/_mapping'

--Jamshid

On Wednesday, April 17, 2013 1:57:14 PM UTC-5, Ivan Brusic wrote:

Dynamic templates is what you are looking for:
Elasticsearch Platform — Find real-time answers at scale | Elastic

Looking for dynamic_templates toward the bottom. That section really
should be a standalone page IMHO.

--
Ivan

On Wed, Apr 17, 2013 at 11:47 AM, Jamshid <jams...@gmail.com <javascript:>

wrote:

Hi, I'm currently setting the type of the "known" fields in my index via
elasticsearch-0.19.3/config/templates/foo.json:

{
"foo_template" : {
"template" : "*",
"settings" : {
"index.number_of_shards" : 3,
"index.number_of_replicas" : 1
},
"mappings" : {
"default" : {
"_source" : { "enabled" : true,
"compress" : true },
"_all" : { "enabled" : false },
"_timestamp" : { "enabled" : true,
"store" : true },
"analyzer" : "keyword",
"properties" : {
"name" : { "type" : "string", "analyzer" : "keyword"
},
"size" : { "type" : "long" }
}
}
}
}
}

Problem is that I also index user-supplied fields, but don't know the
names of the field beforehand. E.g.:

$ curl -XPOST 'http://localhost:9200/test/123' -d '{ "name" : "test", "x-foo-meta"
: "2013-01-01"
, "size" : 1 }'

But I don't want "x-foo-meta" type to ever be date, only string. If the
first insert happens to contain a value that looks like a date then the
mapping for the field is permanently set to date:

$ curl -XGET 'http://localhost:9200/test/_mapping'

{"test":{"123":{"analyzer":"keyword","_all":{"enabled":false},"_timestamp":{"store":"yes","enabled":true},"_source":{"compress":true},"properties":{"name":{"type":"string","analyzer":"keyword"},"size":{"type":"long"},
"x-foo-meta":{"type":"date"
,"format":"dateOptionalTime"}}},"default":{"analyzer":"keyword","_all":{"enabled":false},"_timestamp":{"store":"yes","enabled":true},"_source":{"compress":true},"properties":{"name":{"type":"string","analyzer":"keyword"},"size":{"type":"long"}}}}}

If I try to insert a non-date value I get an error:

$ curl -XPOST 'http://localhost:9200/test/123' -d '{ "name" : "test", "x-foo-meta"
: "x2013-01-01"
, "size" : 1 }'
{"error":"MapperParsingException[Failed to parse [x-foo-meta]]; nested:
MapperParsingException[failed to parse date field [x2013-01-01], tried both
date format [dateOptionalTime], and timestamp number]; nested:
IllegalArgumentException[Invalid format: "x2013-01-01"]; ","status":400}

So, how can I force the type of all unspecified fields to be "string"?
Wildcards don't work in "properties", and setting "type" : "string" in the
foo.json caused an error on insert, I assume because the "size" field I'm
inserting is not a string:

{"error":"MapperParsingException[mapping [default]]; nested:
MapperParsingException[Trying to parse an object but has a different type
[string] for [default]]; ","status":400}

Thanks,
Jamshid

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.