Is it possible to create mapping for fields following a pattern or wildcard?


(Diptamay) #1

Hi Shay

I have documents of this type as listed under https://gist.github.com/661688

I was wondering if I could create mappings for fields that follow a
certain pattern, as against ES determining dynamically how to index,
store etc those fields. As an example, I want all the nested field
values under "tags" section/object in the documents to all be treated
as Strings. So I am looking to define a mapping like the following.
Note the "*" below under "tags".

{
"video" : {
"dynamic" : true,
"enabled" : true,
"date_formats" : [
"dateOptionalTime",
"yyyy/MM/dd HH:mm:ss||yyyy/MM/dd"
],
"_source" : {
"enabled" : true,
"compress" : true,
"name" : "_source"
},
"properties" : {
"itemType" : {
"store" : "yes",
"type" : "string"
},
"userDate" : {
"store" : "yes",
"format" : "dateOptionalTime",
"type" : "date"
},
"tags" : {
"properties" : {
"*" : {
"properties" : {
"value" : {
"type" : "string"
},
"displayName" : {
"type" : "string"
}
},
"type" : "object"
}
}
}
},
"_all" : {
"enabled" : true,
"store" : "no",
"term_vector" : "no"
},
"type" : "object"
}
}

So whenever, a document would have, something like the following tags
object, all the object values listed under combined_media_state/
player_id/team_id, would be treated as strings as against dynamically
decided by ES like integer for player_id.value

"tags" : {
"combined_media_state" : [ {
"value" : "MEDIA_ON",
"displayName" : "MEDIA_ON"
} ],
"weight" : [ {
"value" : "1",
"displayName" : "1"
} ],
"subject" : [ {
"value" : "MLBCOM_FEATURE",
"displayName" : "MLBCOM_FEATURE"
} ],
"mlbtax" : [ {
"value" : "mm_hightlight_reel",
"displayName" : "highlight reel"
}, {
"value" : "home_run",
"displayName" : "home run"
} ],
"player_id" : [ {
"value" : "121347",
"displayName" : "Alex Rodriguez"
} ],
"team_id" : [ {
"value" : "147",
"displayName" : "New York Yankees"
} ]
}

I guess, this is probably not there right now. Would it be possible to
add this as a feature request or is it too much of a hassle?

Thanks
Diptamay


(Shay Banon) #2

There is the dynamic templates feature:
http://www.elasticsearch.com/docs/elasticsearch/mapping/root_object_type/#dynamic_templates,
but it can only be defined on the root object mapping, so no notion of
applying (matching) the template only for specific objects (like tags in
your case). But, it can be easily added by adding, on top of match, also
path_match, which will match on the current object "path", in your case
its tags. You can open a feature for it.

-shay.banon

On Thu, Nov 4, 2010 at 12:57 AM, diptamay diptamay@gmail.com wrote:

Hi Shay

I have documents of this type as listed under
https://gist.github.com/661688

I was wondering if I could create mappings for fields that follow a
certain pattern, as against ES determining dynamically how to index,
store etc those fields. As an example, I want all the nested field
values under "tags" section/object in the documents to all be treated
as Strings. So I am looking to define a mapping like the following.
Note the "*" below under "tags".

{
"video" : {
"dynamic" : true,
"enabled" : true,
"date_formats" : [
"dateOptionalTime",
"yyyy/MM/dd HH:mm:ss||yyyy/MM/dd"
],
"_source" : {
"enabled" : true,
"compress" : true,
"name" : "_source"
},
"properties" : {
"itemType" : {
"store" : "yes",
"type" : "string"
},
"userDate" : {
"store" : "yes",
"format" : "dateOptionalTime",
"type" : "date"
},
"tags" : {
"properties" : {
"*" : {
"properties" : {
"value" : {
"type" : "string"
},
"displayName" : {
"type" : "string"
}
},
"type" : "object"
}
}
}
},
"_all" : {
"enabled" : true,
"store" : "no",
"term_vector" : "no"
},
"type" : "object"
}
}

So whenever, a document would have, something like the following tags
object, all the object values listed under combined_media_state/
player_id/team_id, would be treated as strings as against dynamically
decided by ES like integer for player_id.value

"tags" : {
"combined_media_state" : [ {
"value" : "MEDIA_ON",
"displayName" : "MEDIA_ON"
} ],
"weight" : [ {
"value" : "1",
"displayName" : "1"
} ],
"subject" : [ {
"value" : "MLBCOM_FEATURE",
"displayName" : "MLBCOM_FEATURE"
} ],
"mlbtax" : [ {
"value" : "mm_hightlight_reel",
"displayName" : "highlight reel"
}, {
"value" : "home_run",
"displayName" : "home run"
} ],
"player_id" : [ {
"value" : "121347",
"displayName" : "Alex Rodriguez"
} ],
"team_id" : [ {
"value" : "147",
"displayName" : "New York Yankees"
} ]
}

I guess, this is probably not there right now. Would it be possible to
add this as a feature request or is it too much of a hassle?

Thanks
Diptamay


(Shay Banon) #3

Added path_match to dynamic_template:
https://github.com/elasticsearch/elasticsearch/issues/issue/476.

On Thu, Nov 4, 2010 at 1:20 AM, Shay Banon shay.banon@elasticsearch.comwrote:

There is the dynamic templates feature:
http://www.elasticsearch.com/docs/elasticsearch/mapping/root_object_type/#dynamic_templates,
but it can only be defined on the root object mapping, so no notion of
applying (matching) the template only for specific objects (like tags in
your case). But, it can be easily added by adding, on top of match, also
path_match, which will match on the current object "path", in your case
its tags. You can open a feature for it.

-shay.banon

On Thu, Nov 4, 2010 at 12:57 AM, diptamay diptamay@gmail.com wrote:

Hi Shay

I have documents of this type as listed under
https://gist.github.com/661688

I was wondering if I could create mappings for fields that follow a
certain pattern, as against ES determining dynamically how to index,
store etc those fields. As an example, I want all the nested field
values under "tags" section/object in the documents to all be treated
as Strings. So I am looking to define a mapping like the following.
Note the "*" below under "tags".

{
"video" : {
"dynamic" : true,
"enabled" : true,
"date_formats" : [
"dateOptionalTime",
"yyyy/MM/dd HH:mm:ss||yyyy/MM/dd"
],
"_source" : {
"enabled" : true,
"compress" : true,
"name" : "_source"
},
"properties" : {
"itemType" : {
"store" : "yes",
"type" : "string"
},
"userDate" : {
"store" : "yes",
"format" : "dateOptionalTime",
"type" : "date"
},
"tags" : {
"properties" : {
"*" : {
"properties" : {
"value" : {
"type" : "string"
},
"displayName" : {
"type" : "string"
}
},
"type" : "object"
}
}
}
},
"_all" : {
"enabled" : true,
"store" : "no",
"term_vector" : "no"
},
"type" : "object"
}
}

So whenever, a document would have, something like the following tags
object, all the object values listed under combined_media_state/
player_id/team_id, would be treated as strings as against dynamically
decided by ES like integer for player_id.value

"tags" : {
"combined_media_state" : [ {
"value" : "MEDIA_ON",
"displayName" : "MEDIA_ON"
} ],
"weight" : [ {
"value" : "1",
"displayName" : "1"
} ],
"subject" : [ {
"value" : "MLBCOM_FEATURE",
"displayName" : "MLBCOM_FEATURE"
} ],
"mlbtax" : [ {
"value" : "mm_hightlight_reel",
"displayName" : "highlight reel"
}, {
"value" : "home_run",
"displayName" : "home run"
} ],
"player_id" : [ {
"value" : "121347",
"displayName" : "Alex Rodriguez"
} ],
"team_id" : [ {
"value" : "147",
"displayName" : "New York Yankees"
} ]
}

I guess, this is probably not there right now. Would it be possible to
add this as a feature request or is it too much of a hassle?

Thanks
Diptamay


(Diptamay) #4

Awesome. Thanks!

On Nov 4, 4:55 am, Shay Banon shay.ba...@elasticsearch.com wrote:

Added path_match to dynamic_template:https://github.com/elasticsearch/elasticsearch/issues/issue/476.

On Thu, Nov 4, 2010 at 1:20 AM, Shay Banon shay.ba...@elasticsearch.comwrote:

There is the dynamic templates feature:
http://www.elasticsearch.com/docs/elasticsearch/mapping/root_object_t...,
but it can only be defined on the root object mapping, so no notion of
applying (matching) the template only for specific objects (like tags in
your case). But, it can be easily added by adding, on top of match, also
path_match, which will match on the current object "path", in your case
its tags. You can open a feature for it.

-shay.banon

On Thu, Nov 4, 2010 at 12:57 AM, diptamay dipta...@gmail.com wrote:

Hi Shay

I have documents of this type as listed under
https://gist.github.com/661688

I was wondering if I could create mappings for fields that follow a
certain pattern, as against ES determining dynamically how to index,
store etc those fields. As an example, I want all the nested field
values under "tags" section/object in the documents to all be treated
as Strings. So I am looking to define a mapping like the following.
Note the "*" below under "tags".

{
"video" : {
"dynamic" : true,
"enabled" : true,
"date_formats" : [
"dateOptionalTime",
"yyyy/MM/dd HH:mm:ss||yyyy/MM/dd"
],
"_source" : {
"enabled" : true,
"compress" : true,
"name" : "_source"
},
"properties" : {
"itemType" : {
"store" : "yes",
"type" : "string"
},
"userDate" : {
"store" : "yes",
"format" : "dateOptionalTime",
"type" : "date"
},
"tags" : {
"properties" : {
"*" : {
"properties" : {
"value" : {
"type" : "string"
},
"displayName" : {
"type" : "string"
}
},
"type" : "object"
}
}
}
},
"_all" : {
"enabled" : true,
"store" : "no",
"term_vector" : "no"
},
"type" : "object"
}
}

So whenever, a document would have, something like the following tags
object, all the object values listed under combined_media_state/
player_id/team_id, would be treated as strings as against dynamically
decided by ES like integer for player_id.value

"tags" : {
"combined_media_state" : [ {
"value" : "MEDIA_ON",
"displayName" : "MEDIA_ON"
} ],
"weight" : [ {
"value" : "1",
"displayName" : "1"
} ],
"subject" : [ {
"value" : "MLBCOM_FEATURE",
"displayName" : "MLBCOM_FEATURE"
} ],
"mlbtax" : [ {
"value" : "mm_hightlight_reel",
"displayName" : "highlight reel"
}, {
"value" : "home_run",
"displayName" : "home run"
} ],
"player_id" : [ {
"value" : "121347",
"displayName" : "Alex Rodriguez"
} ],
"team_id" : [ {
"value" : "147",
"displayName" : "New York Yankees"
} ]
}

I guess, this is probably not there right now. Would it be possible to
add this as a feature request or is it too much of a hassle?

Thanks
Diptamay


(system) #5