Mappings for Nested Objects

[drendell (Darrel Rendell)]

Hi all,

I'm having trouble building out a suitable mapping structure that will appropriately accommodate nested objects.

ES and Kibana are both the latest versions.

Here's a truncated example (I've trimmed the two nested blocks down to reasonable sizes):

{  
   "id":12345,
   "label":"A string of text",
   "published_by":"some user id",
   "is_published":"boolean",
   "last_published":"2018-06-14",
   "first_published":"2018-06-14",
   "brand":"A keyword",
   "language":"ISO country code",
   "sender_ips":[  
      {  
         "ip":"1.2.3.4",
         "country":"England",
         "city":"London"
      },
      {  
         "ip":"1.2.1.2",
         "country":"Germany",
         "city":"Ravensburg"
      },
      {  
         "ip":"1.2.3.1",
         "country":"United States",
         "city":"West Lafayette"
      }
   ],
   "block_data":[  
      {  
         "data":"A Keyword",
         "impact":"A Keyword",
         "family":"A Keyword",
         "phenotype":"A keyword"
      },
      {  
         "data":"A Keyword",
         "impact":"A Keyword",
         "family":"A Keyword",
         "phenotype":"A keyword"
      },
      {  
         "data":"A Keyword",
         "impact":"A Keyword",
         "family":"A Keyword",
         "phenotype":"A keyword"
      }
        ],
   "m_ids":[  
      111111111,
      222222222,
      333333333
   ]
}

The mappings I'm currently using:

{
  "mappings": {
    "_doc": {
      "properties": {
        "id": {"type": "keyword"},
        "label": {"type": "text"},
		"published_by": {"type": "keyword"},
		"is_published": {"type": "boolean"},
		"first_published": {"type": "date"},
		"last_published": {"type": "date"},
		"brand": {"type": "keyword"},
		"language": {"type": "keyword"},
		"block_data": {
			"type": "nested",
			"properties": {
				"data": {"type": "keyword"},
				"impact": {"type": "keyword"},
				"family": {"type": "keyword"},
				"phenotype": {"type": "keyword"}
			}
			
		},
		"sender_ips": {
			"type": "nested",
			"properties": {
				"ip": {"type": "ip"},
				"country": {"type": "keyword"},
				"city": {"type": "keyword"}
			}			
		},
		"m_id": {"type": "text"}		
      }
    }
  }
}

The challenge I have, is that the sender_ips fields do not populate (the block_data fields seem to work fine).

Kibana explicitly warns that nested objects are not well supported:

Am I approaching this incorrectly? Is there a better way to think about storing these types of objects?

Cheers, for any help!

[Mark_Harwood]

"Nested" is only required if:

1. you need to test >1 property of an object in a query or,
2. you need to count the number of nested objects rather than the number of root level docs

An example of problem 1) is if your queries are looking for city:Paris AND country:France - if you don't use nested then you could mistakenly match a doc that includes these 2 IP objects:

[  { city:Lyon, country:France}, {city:Paris, country:US}]

So Paris, Texas would match here because of what we call "cross-matching".
If you only ever query for a single attribute eg city:France then there is no problem and you can avoid the need for nested.