How to create mapping that deals with variable types in fields

Elastic Stack Elasticsearch
1

I have a situation in which the documents being indexed occasionally have
fields with different types. I don't control the documents being indexed
and sometimes they will have field that, when first seen, is a number type,
but in later documents may be a string type.

I've tried to create a dynamic_mapping to treat everything as a string, but
it doesn't appear to be working for me.

The following sequence will fail (note the "test" field changes type):

curl -XDELETE "http://localhost:9200/test/"

curl -XPUT "http://localhost:9200/test/"

curl -XPUT http://localhost:9200/test/test/_mapping -d '
{
"test": {
"dynamic_templates": [
{
"all_fields": {
"match": "*",
"match_mapping_type" : ["string", "integer", "long", "float",
"double", "boolean"],
"mapping": {
"type": "string"
}
}
}
],
"properties": {
"meta": {
"type": "object",
"properties": {
"timestamp": {
"store": "yes",
"type": "date"
},
"foo": {
"type": "object",
"properties": {
"bar": {
"index": "not_analyzed",
"type": "string"
}
}
}
}
}
}
}
}'

curl -XPUT http://localhost:9200/test/test/1 -d '
{
"meta": [{
"timestamp": 1234567890,
"test": 123,
"foo": {
"bar" : "Hello World"
},
"fruits" : {
"canned": {
"orchard" : ["Apples", "Oranges", "Peaches"]
}
}
}]
}'

curl -XPUT http://localhost:9200/test/test/1 -d '
{
"meta": [{
"timestamp": 1234567890,
"test": "abc",
"foo": {
"bar" : "Hello World"
},
"fruits" : {
"canned": {
"orchard" : ["Apples", "Oranges", "Peaches"]
}
}
}]
}'

The error reported is:

{
"error": "MapperParsingException[Failed to parse [meta.test]]; nested:
NumberFormatException[For input string: "abc"]; ",
"status": 400
}

Anyone know how I create a mapping to deal with this case?

Thanks.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Also the following fails despite the dynamic_template in the mapping:

curl -XDELETE "http://localhost:9200/test/"

curl -XPUT "http://localhost:9200/test/"

curl -XPUT http://localhost:9200/test/test/_mapping -d '
{
"test": {
"date_detection" : false,
"numeric_detection": false,
"dynamic_templates": [
{
"all_fields": {
"match": "*",
"match_mapping_type" : "integer",
"mapping": {
"type": "string"
}
}
}
]
}
}'

curl -XPUT http://localhost:9200/test/test/1 -d '
{
"test": 123,
}'

curl -XPUT http://localhost:9200/test/test/2 -d '
{
"test": "abc",
}'

Results in:

java.lang.NumberFormatException: For input string: "abc"
at java.lang.NumberFormatException.forInputString(NumberFormatException.
java:48)
at java.lang.Long.parseLong(Long.java:410)
at java.lang.Long.parseLong(Long.java:468)
at org.elasticsearch.common.xcontent.support.AbstractXContentParser.
longValue(AbstractXContentParser.java:72)
at org.elasticsearch.index.mapper.core.LongFieldMapper.
innerParseCreateField(LongFieldMapper.java:284)
at org.elasticsearch.index.mapper.core.NumberFieldMapper.parseCreateField(
NumberFieldMapper.java:182)
at org.elasticsearch.index.mapper.core.AbstractFieldMapper.parse(
AbstractFieldMapper.java:307)
... 11 more

On Tuesday, April 16, 2013 10:34:57 AM UTC-7, Jason wrote:

I have a situation in which the documents being indexed occasionally have
fields with different types. I don't control the documents being indexed
and sometimes they will have field that, when first seen, is a number type,
but in later documents may be a string type.

I've tried to create a dynamic_mapping to treat everything as a string,
but it doesn't appear to be working for me.

The following sequence will fail (note the "test" field changes type):

curl -XDELETE "http://localhost:9200/test/"

curl -XPUT "http://localhost:9200/test/"

curl -XPUT http://localhost:9200/test/test/_mapping -d '
{
"test": {
"dynamic_templates": [
{
"all_fields": {
"match": "*",
"match_mapping_type" : ["string", "integer", "long", "float",
"double", "boolean"],
"mapping": {
"type": "string"
}
}
}
],
"properties": {
"meta": {
"type": "object",
"properties": {
"timestamp": {
"store": "yes",
"type": "date"
},
"foo": {
"type": "object",
"properties": {
"bar": {
"index": "not_analyzed",
"type": "string"
}
}
}
}
}
}
}
}'

curl -XPUT http://localhost:9200/test/test/1 -d '
{
"meta": [{
"timestamp": 1234567890,
"test": 123,
"foo": {
"bar" : "Hello World"
},
"fruits" : {
"canned": {
"orchard" : ["Apples", "Oranges", "Peaches"]
}
}
}]
}'

curl -XPUT http://localhost:9200/test/test/1 -d '
{
"meta": [{
"timestamp": 1234567890,
"test": "abc",
"foo": {
"bar" : "Hello World"
},
"fruits" : {
"canned": {
"orchard" : ["Apples", "Oranges", "Peaches"]
}
}
}]
}'

The error reported is:

{
"error": "MapperParsingException[Failed to parse [meta.test]];
nested: NumberFormatException[For input string: "abc"]; ",
"status": 400
}

Anyone know how I create a mapping to deal with this case?

Thanks.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

My bad.. the last post does work.. the mapping needs to be type long:

curl -XPUT http://localhost:9200/test/test/_mapping -d '
{
"test": {
"date_detection" : false,
"numeric_detection": false,
"dynamic_templates": [
{
"all_fields": {
"match": "*",
"match_mapping_type" : "long",
"mapping": {
"type": "string"
}
}
}
]
}
}'

On Tuesday, April 16, 2013 10:34:57 AM UTC-7, Jason wrote:

I have a situation in which the documents being indexed occasionally have
fields with different types. I don't control the documents being indexed
and sometimes they will have field that, when first seen, is a number type,
but in later documents may be a string type.

I've tried to create a dynamic_mapping to treat everything as a string,
but it doesn't appear to be working for me.

The following sequence will fail (note the "test" field changes type):

curl -XDELETE "http://localhost:9200/test/"

curl -XPUT "http://localhost:9200/test/"

curl -XPUT http://localhost:9200/test/test/_mapping -d '
{
"test": {
"dynamic_templates": [
{
"all_fields": {
"match": "*",
"match_mapping_type" : ["string", "integer", "long", "float",
"double", "boolean"],
"mapping": {
"type": "string"
}
}
}
],
"properties": {
"meta": {
"type": "object",
"properties": {
"timestamp": {
"store": "yes",
"type": "date"
},
"foo": {
"type": "object",
"properties": {
"bar": {
"index": "not_analyzed",
"type": "string"
}
}
}
}
}
}
}
}'

curl -XPUT http://localhost:9200/test/test/1 -d '
{
"meta": [{
"timestamp": 1234567890,
"test": 123,
"foo": {
"bar" : "Hello World"
},
"fruits" : {
"canned": {
"orchard" : ["Apples", "Oranges", "Peaches"]
}
}
}]
}'

curl -XPUT http://localhost:9200/test/test/1 -d '
{
"meta": [{
"timestamp": 1234567890,
"test": "abc",
"foo": {
"bar" : "Hello World"
},
"fruits" : {
"canned": {
"orchard" : ["Apples", "Oranges", "Peaches"]
}
}
}]
}'

The error reported is:

{
"error": "MapperParsingException[Failed to parse [meta.test]];
nested: NumberFormatException[For input string: "abc"]; ",
"status": 400
}

Anyone know how I create a mapping to deal with this case?

Thanks.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.