Is it possible to create mapping for fields following a pattern or wildcard?

Diptamay · November 3, 2010, 10:57pm

Hi Shay

I have documents of this type as listed under https://gist.github.com/661688

I was wondering if I could create mappings for fields that follow a
certain pattern, as against ES determining dynamically how to index,
store etc those fields. As an example, I want all the nested field
values under "tags" section/object in the documents to all be treated
as Strings. So I am looking to define a mapping like the following.
Note the "*" below under "tags".

{
"video" : {
"dynamic" : true,
"enabled" : true,
"date_formats" : [
"dateOptionalTime",
"yyyy/MM/dd HH:mm:ss||yyyy/MM/dd"
],
"_source" : {
"enabled" : true,
"compress" : true,
"name" : "_source"
},
"properties" : {
"itemType" : {
"store" : "yes",
"type" : "string"
},
"userDate" : {
"store" : "yes",
"format" : "dateOptionalTime",
"type" : "date"
},
"tags" : {
"properties" : {
"*" : {
"properties" : {
"value" : {
"type" : "string"
},
"displayName" : {
"type" : "string"
}
},
"type" : "object"
}
}
}
},
"_all" : {
"enabled" : true,
"store" : "no",
"term_vector" : "no"
},
"type" : "object"
}
}

So whenever, a document would have, something like the following tags
object, all the object values listed under combined_media_state/
player_id/team_id, would be treated as strings as against dynamically
decided by ES like integer for player_id.value

"tags" : {
"combined_media_state" : [ {
"value" : "MEDIA_ON",
"displayName" : "MEDIA_ON"
} ],
"weight" : [ {
"value" : "1",
"displayName" : "1"
} ],
"subject" : [ {
"value" : "MLBCOM_FEATURE",
"displayName" : "MLBCOM_FEATURE"
} ],
"mlbtax" : [ {
"value" : "mm_hightlight_reel",
"displayName" : "highlight reel"
}, {
"value" : "home_run",
"displayName" : "home run"
} ],
"player_id" : [ {
"value" : "121347",
"displayName" : "Alex Rodriguez"
} ],
"team_id" : [ {
"value" : "147",
"displayName" : "New York Yankees"
} ]
}

I guess, this is probably not there right now. Would it be possible to
add this as a feature request or is it too much of a hassle?

Thanks
Diptamay

kimchy · November 3, 2010, 11:20pm

There is the dynamic templates feature:
http://www.elasticsearch.com/docs/elasticsearch/mapping/root_object_type/#dynamic_templates,
but it can only be defined on the root object mapping, so no notion of
applying (matching) the template only for specific objects (like tags in
your case). But, it can be easily added by adding, on top of match, also
path_match, which will match on the current object "path", in your case
its tags. You can open a feature for it.

-shay.banon

On Thu, Nov 4, 2010 at 12:57 AM, diptamay diptamay@gmail.com wrote:

Hi Shay

I have documents of this type as listed under
661688’s gists · GitHub

I was wondering if I could create mappings for fields that follow a
certain pattern, as against ES determining dynamically how to index,
store etc those fields. As an example, I want all the nested field
values under "tags" section/object in the documents to all be treated
as Strings. So I am looking to define a mapping like the following.
Note the "*" below under "tags".

{
"video" : {
"dynamic" : true,
"enabled" : true,
"date_formats" : [
"dateOptionalTime",
"yyyy/MM/dd HH:mm:ss||yyyy/MM/dd"
],
"_source" : {
"enabled" : true,
"compress" : true,
"name" : "_source"
},
"properties" : {
"itemType" : {
"store" : "yes",
"type" : "string"
},
"userDate" : {
"store" : "yes",
"format" : "dateOptionalTime",
"type" : "date"
},
"tags" : {
"properties" : {
"*" : {
"properties" : {
"value" : {
"type" : "string"
},
"displayName" : {
"type" : "string"
}
},
"type" : "object"
}
}
}
},
"_all" : {
"enabled" : true,
"store" : "no",
"term_vector" : "no"
},
"type" : "object"
}
}

So whenever, a document would have, something like the following tags
object, all the object values listed under combined_media_state/
player_id/team_id, would be treated as strings as against dynamically
decided by ES like integer for player_id.value

"tags" : {
"combined_media_state" : [ {
"value" : "MEDIA_ON",
"displayName" : "MEDIA_ON"
} ],
"weight" : [ {
"value" : "1",
"displayName" : "1"
} ],
"subject" : [ {
"value" : "MLBCOM_FEATURE",
"displayName" : "MLBCOM_FEATURE"
} ],
"mlbtax" : [ {
"value" : "mm_hightlight_reel",
"displayName" : "highlight reel"
}, {
"value" : "home_run",
"displayName" : "home run"
} ],
"player_id" : [ {
"value" : "121347",
"displayName" : "Alex Rodriguez"
} ],
"team_id" : [ {
"value" : "147",
"displayName" : "New York Yankees"
} ]
}

I guess, this is probably not there right now. Would it be possible to
add this as a feature request or is it too much of a hassle?

Thanks
Diptamay

kimchy · November 4, 2010, 8:55am

Added path_match to dynamic_template:
Mapper: Add `path_match` for full object navigation path matching · Issue #476 · elastic/elasticsearch · GitHub.

On Thu, Nov 4, 2010 at 1:20 AM, Shay Banon shay.banon@elasticsearch.comwrote:

There is the dynamic templates feature:
http://www.elasticsearch.com/docs/elasticsearch/mapping/root_object_type/#dynamic_templates,
but it can only be defined on the root object mapping, so no notion of
applying (matching) the template only for specific objects (like tags in
your case). But, it can be easily added by adding, on top of match, also
path_match, which will match on the current object "path", in your case
its tags. You can open a feature for it.

-shay.banon

On Thu, Nov 4, 2010 at 12:57 AM, diptamay diptamay@gmail.com wrote:

Hi Shay

I have documents of this type as listed under
661688’s gists · GitHub

I was wondering if I could create mappings for fields that follow a
certain pattern, as against ES determining dynamically how to index,
store etc those fields. As an example, I want all the nested field
values under "tags" section/object in the documents to all be treated
as Strings. So I am looking to define a mapping like the following.
Note the "*" below under "tags".

{
"video" : {
"dynamic" : true,
"enabled" : true,
"date_formats" : [
"dateOptionalTime",
"yyyy/MM/dd HH:mm:ss||yyyy/MM/dd"
],
"_source" : {
"enabled" : true,
"compress" : true,
"name" : "_source"
},
"properties" : {
"itemType" : {
"store" : "yes",
"type" : "string"
},
"userDate" : {
"store" : "yes",
"format" : "dateOptionalTime",
"type" : "date"
},
"tags" : {
"properties" : {
"*" : {
"properties" : {
"value" : {
"type" : "string"
},
"displayName" : {
"type" : "string"
}
},
"type" : "object"
}
}
}
},
"_all" : {
"enabled" : true,
"store" : "no",
"term_vector" : "no"
},
"type" : "object"
}
}

So whenever, a document would have, something like the following tags
object, all the object values listed under combined_media_state/
player_id/team_id, would be treated as strings as against dynamically
decided by ES like integer for player_id.value

"tags" : {
"combined_media_state" : [ {
"value" : "MEDIA_ON",
"displayName" : "MEDIA_ON"
} ],
"weight" : [ {
"value" : "1",
"displayName" : "1"
} ],
"subject" : [ {
"value" : "MLBCOM_FEATURE",
"displayName" : "MLBCOM_FEATURE"
} ],
"mlbtax" : [ {
"value" : "mm_hightlight_reel",
"displayName" : "highlight reel"
}, {
"value" : "home_run",
"displayName" : "home run"
} ],
"player_id" : [ {
"value" : "121347",
"displayName" : "Alex Rodriguez"
} ],
"team_id" : [ {
"value" : "147",
"displayName" : "New York Yankees"
} ]
}

I guess, this is probably not there right now. Would it be possible to
add this as a feature request or is it too much of a hassle?

Thanks
Diptamay

Diptamay · November 4, 2010, 3:36pm

Awesome. Thanks!

On Nov 4, 4:55 am, Shay Banon shay.ba...@elasticsearch.com wrote:

Added path_match to dynamic_template:Mapper: Add `path_match` for full object navigation path matching · Issue #476 · elastic/elasticsearch · GitHub.

On Thu, Nov 4, 2010 at 1:20 AM, Shay Banon shay.ba...@elasticsearch.comwrote:

There is the dynamic templates feature:
http://www.elasticsearch.com/docs/elasticsearch/mapping/root_object_t...,
but it can only be defined on the root object mapping, so no notion of
applying (matching) the template only for specific objects (like tags in
your case). But, it can be easily added by adding, on top of match, also
path_match, which will match on the current object "path", in your case
its tags. You can open a feature for it.

-shay.banon

On Thu, Nov 4, 2010 at 12:57 AM, diptamay dipta...@gmail.com wrote:

Hi Shay

I have documents of this type as listed under
661688’s gists · GitHub

I was wondering if I could create mappings for fields that follow a
certain pattern, as against ES determining dynamically how to index,
store etc those fields. As an example, I want all the nested field
values under "tags" section/object in the documents to all be treated
as Strings. So I am looking to define a mapping like the following.
Note the "*" below under "tags".

{
"video" : {
"dynamic" : true,
"enabled" : true,
"date_formats" : [
"dateOptionalTime",
"yyyy/MM/dd HH:mm:ss||yyyy/MM/dd"
],
"_source" : {
"enabled" : true,
"compress" : true,
"name" : "_source"
},
"properties" : {
"itemType" : {
"store" : "yes",
"type" : "string"
},
"userDate" : {
"store" : "yes",
"format" : "dateOptionalTime",
"type" : "date"
},
"tags" : {
"properties" : {
"*" : {
"properties" : {
"value" : {
"type" : "string"
},
"displayName" : {
"type" : "string"
}
},
"type" : "object"
}
}
}
},
"_all" : {
"enabled" : true,
"store" : "no",
"term_vector" : "no"
},
"type" : "object"
}
}

So whenever, a document would have, something like the following tags
object, all the object values listed under combined_media_state/
player_id/team_id, would be treated as strings as against dynamically
decided by ES like integer for player_id.value

"tags" : {
"combined_media_state" : [ {
"value" : "MEDIA_ON",
"displayName" : "MEDIA_ON"
} ],
"weight" : [ {
"value" : "1",
"displayName" : "1"
} ],
"subject" : [ {
"value" : "MLBCOM_FEATURE",
"displayName" : "MLBCOM_FEATURE"
} ],
"mlbtax" : [ {
"value" : "mm_hightlight_reel",
"displayName" : "highlight reel"
}, {
"value" : "home_run",
"displayName" : "home run"
} ],
"player_id" : [ {
"value" : "121347",
"displayName" : "Alex Rodriguez"
} ],
"team_id" : [ {
"value" : "147",
"displayName" : "New York Yankees"
} ]
}

I guess, this is probably not there right now. Would it be possible to
add this as a feature request or is it too much of a hassle?

Thanks
Diptamay