How to map a dynamic map of key values?


(Drew) #1

Hey Guys,

How can I map an arbitrary map of key/values in ES? My JSON looks like the following, where "name" and "age" are static but "attributes" is dynamic:

{
"name": "john",
"age": 25,
"attributes" : {
"key1": value1,
"key2": value2,
"key3": value3,
...
}
}

Things to consider:

  1. Not all documents have the same number of attributes or even the same attributes.
  2. Different documents can have values of different types for the same name attributes (say "attr1" is string for doc 1 but is int for doc 2)
  3. Attribute value can only be primitive json type (boolean, integer, number, null or string) or an array of primitive type.
  4. It goes without saying that the attributes must be searchable.

Thanks,

Drew

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/BB50D9C2-1F16-437F-A6AB-7FEB25427743%40venarc.com.
For more options, visit https://groups.google.com/d/optout.


(Thomas) #2

one way would be to use a nested document structure like:
check: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-nested-type.html#mapping-nested-type
..
"properties" : {
"type":"nested"
"attributes":{
"key" : {
"index" : "not_analyzed",
"type" : "string"
},
"value" : {
"index":"not_analyzed"
"type" : "string"
}
}
}
...
You may even use multiple fields depending of the type of the value, and if
you know the type of each key to be able to index it properly and do search
upon it

"properties" : {
"type":"nested"
"attributes":{
"key" : {
"index" : "not_analyzed",
"type" : "string"
},
"stringValue" : {
"index":"not_analyzed"
"type" : "string"
},
"numericValue" : {
"type" : "float"
}
}
}

How are you going to update the document? If you chose this way and you
perform updates on a nested document(e.g. add a key value) the whole
document is reindexed

Hope it helps

Thomas

On Wednesday, 11 June 2014 04:09:16 UTC+3, Drew wrote:

Hey Guys,

How can I map an arbitrary map of key/values in ES? My JSON looks like the
following, where “name" and “age” are static but “attributes” is dynamic:

{
“name”: “john”,
“age”: 25,
“attributes” : {
“key1”: value1,
“key2”: value2,
“key3”: value3,
...
}
}

Things to consider:

  1. Not all documents have the same number of attributes or even the same
    attributes.
  2. Different documents can have values of different types for the same
    name attributes (say “attr1” is string for doc 1 but is int for doc 2)
  3. Attribute value can only be primitive json type (boolean, integer,
    number, null or string) or an array of primitive type.
  4. It goes without saying that the attributes must be searchable.

Thanks,

Drew

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fb125974-b7a8-461b-9701-fea71cf38855%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Drew) #3

Thanks. I was also looking at dynamic mapping templates (http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-root-object-type.html#_dynamic_templates) too. Would that approach work if I were to flatten my JSON so it'd be more like:

{
"name": "john",
"age": 25,
"attribute-key1" : value1,
"attribute-key2" : value2,
"attribute-key3" : value3,
}

I could even take this further and make the key/value pair multi-fields so they can contain multiple types either at application side or using (http://www.elasticsearch.org/guide/en/elasticsearch/reference/0.90/mapping-multi-field-type.html)

This is how the JSON will look like if I were to implement the multi-fields application side:
{
"name": "john",
"age": 25,
"attribute-string-key1" : value1,
"attribute-int-key2" : value2,
"attribute-string-key3" : value3,
}

Do you see any pro/cons with this approach vs the nested types approach?

  • Drew

On Jun 11, 2014, at 1:39 AM, Thomas thomas.bolis@gmail.com wrote:

one way would be to use a nested document structure like:
check: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-nested-type.html#mapping-nested-type
..
"properties" : {
"type":"nested"
"attributes":{
"key" : {
"index" : "not_analyzed",
"type" : "string"
},
"value" : {
"index":"not_analyzed"
"type" : "string"
}
}
}
...
You may even use multiple fields depending of the type of the value, and if you know the type of each key to be able to index it properly and do search upon it

"properties" : {
"type":"nested"
"attributes":{
"key" : {
"index" : "not_analyzed",
"type" : "string"
},
"stringValue" : {
"index":"not_analyzed"
"type" : "string"
},
"numericValue" : {
"type" : "float"
}
}
}

How are you going to update the document? If you chose this way and you perform updates on a nested document(e.g. add a key value) the whole document is reindexed

Hope it helps

Thomas

On Wednesday, 11 June 2014 04:09:16 UTC+3, Drew wrote:
Hey Guys,

How can I map an arbitrary map of key/values in ES? My JSON looks like the following, where "name" and "age" are static but "attributes" is dynamic:

{
"name": "john",
"age": 25,
"attributes" : {
"key1": value1,
"key2": value2,
"key3": value3,
...
}
}

Things to consider:

  1. Not all documents have the same number of attributes or even the same attributes.
  2. Different documents can have values of different types for the same name attributes (say "attr1" is string for doc 1 but is int for doc 2)
  3. Attribute value can only be primitive json type (boolean, integer, number, null or string) or an array of primitive type.
  4. It goes without saying that the attributes must be searchable.

Thanks,

Drew

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fb125974-b7a8-461b-9701-fea71cf38855%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/688F6074-9B85-4769-99BD-706EEE8903B3%40venarc.com.
For more options, visit https://groups.google.com/d/optout.


(system) #4