Sample data (kibana_sample_data_ecommerce) Question

[sung_il_Kim]

Hello everyone.
I'm a newbie in Korea.

I've some questions about nested objects.
Oh, I already know that Kibana doesn't support nested objects yet.

But I'm curious about sample data - 'kibana_sample_data_ecommerce' on Elastic cloud

There is a key (products) which has an array of the JSON type like below.

{
  "_index": "kibana_sample_data_ecommerce",
  "_type": "_doc",
  "_id": "IvjrrG8BD9puQIJ7dy9v",
  "_version": 1,
  "_score": 0,
  "_source": {
    "category": [
      "Men's Clothing"
    ],
    "currency": "EUR",
    "customer_first_name": "Oliver",
    "customer_full_name": "Oliver Rios",
    "customer_gender": "MALE",
    "customer_id": 7,
    "customer_last_name": "Rios",
    "customer_phone": "",
    "day_of_week": "Monday",
    "day_of_week_i": 0,
    "email": "oliver@rios-family.zzz",
    "manufacturer": [
      "Low Tide Media",
      "Elitelligence"
    ],
    "order_date": "2020-01-13T09:27:22+00:00",
    "order_id": 565855,
    *"products"*: [
      {
        "base_price": 20.99,
        "discount_percentage": 0,
        "quantity": 1,
        "manufacturer": "Low Tide Media",
        "tax_amount": 0,
        "product_id": 19919,
        "category": "Men's Clothing",
        "sku": "ZO0417504175",
        "taxless_price": 20.99,
        "unit_discount_amount": 0,
        "min_price": 9.87,
        "_id": "sold_product_565855_19919",
        "discount_amount": 0,
        "created_on": "2016-12-12T09:27:22+00:00",
        "product_name": "Shirt - dark blue white",
        "price": 20.99,
        "taxful_price": 20.99,
        "base_unit_price": 20.99
      },
      {
        "base_price": 24.99,
        "discount_percentage": 0,
        "quantity": 1,
        "manufacturer": "Elitelligence",
        "tax_amount": 0,
        "product_id": 24502,
        "category": "Men's Clothing",
        "sku": "ZO0535205352",
        "taxless_price": 24.99,
        "unit_discount_amount": 0,
        "min_price": 12.49,
        "_id": "sold_product_565855_24502",
        "discount_amount": 0,
        "created_on": "2016-12-12T09:27:22+00:00",
        "product_name": "Slim fit jeans - raw blue",
        "price": 24.99,
        "taxful_price": 24.99,
        "base_unit_price": 24.99
      }
    ],
    "sku": [
      "ZO0417504175",
      "ZO0535205352"
    ],
    "taxful_total_price": 45.98,
    "taxless_total_price": 45.98,
    "total_quantity": 2,
    "total_unique_products": 2,
    "type": "order",
    "user": "oliver",
    "geoip": {
      "country_iso_code": "GB",
      "location": {
        "lon": -0.1,
        "lat": 51.5
      },
      "continent_name": "Europe"
    }
  },
  "fields": {
    "order_date": [
      "2020-01-13T09:27:22.000Z"
    ],
    "products.created_on": [
      "2016-12-12T09:27:22.000Z",
      "2016-12-12T09:27:22.000Z"
    ]
  }
}

but I found that "products" is not mapped to the nested type

...
"products": {
        "properties": {
          "_id": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "base_price": {
            "type": "half_float"
          },
          "base_unit_price": {
            "type": "half_float"
          },
          "category": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword"
              }
            }
          },
...

Even though that "Products" is not mapped to the nested type, there is a sample 'Cloud text' in Visualize named "[eCommerce] Top Selling Products' which shows counting numbers of products and sorting it.

and it returns the counted(aggregated) numbers of each product name very well.

Q1. As I know, it needs to be mapped to "nested type" to aggregate over the list. Am I wrong? but How does it works well?

I've tried to put my new sample data like 'kibana_sample_data_ecommerce' using the elasticsearch API. (in dev tool)

The data mapped automatically and everything is okay except visualize functions.
I'm trying to show exactly the same Cloud text but it is not working properly.

The thing what I expected is
Shirt - dark blue white : 5
Slim fit jeans - raw blue : 2

but Kibana returns
Shirt - dark blue white : 2
Slim fit jeans - raw blue : 2

I think it returns the record numbers of that index. Because nested object("products") is not mapped properly.
Also, I can see that docs count is "2" on my new sample data(in index management page) even if the products are 7 rows

Of course, It works very well when I set "nested type" on that field, but Kibana is not supporting it

Q. I'd like to show the cloud text chart using Kibana like 'kibana_sample_data_ecommerce. What should I do?

[sung_il_Kim]

@bhavyarm please give me advice above issue.

[flash1293 (Joe Reuter)]

Hi

Q1. As I know, it needs to be mapped to "nested type" to aggregate over the list. Am I wrong? but How does it works well?

If a field is not mapped to the nested type and you ingest an array, Elasticsearch will flatten the data. That means the following document

{
  key1: 1,
  nested: [
    { nestedKey1: 2, nestedKey2: 3,  },
    { nestedKey1: 4, nestedKey2: 5,  },
    { nestedKey1: 6, nestedKey2: 7  },
  ]
}

becomes

{
  key1: 1,
  nested.nestedKey1: [2, 4, 6],
  nested.nestedKey2: [3, 5, 7],
}

Now you can aggregate over the values in the arrays, but the thing you want to do doesn't work anymore - because the association between the keys of the individual nested objects is lost. The "count" on "Shirt - dark blue white" means the number of the flattened documents that contain "Shirt - dark blue white".

Q. I'd like to show the cloud text chart using Kibana like 'kibana_sample_data_ecommerce. What should I do?

You can't do that without changing the shape of your data. To do aggregations like you want them, you have to denormalize the data and ingest individual documents for the nested documents. In the example above, instead having one document with the nested array, you have three separate documents like this:

{ key1: 1, nestedKey1: 2, nestedKey2: 3 },
{ key1: 1, nestedKey1: 4, nestedKey2: 5 },
{ key1: 1, nestedKey1: 6, nestedKey2: 7 }

Then Elasticsearch knows about the connection between the values of nestedKey1 and nestedKey2 and the counts will give you what you want to see.