Kibana CHART is approximating NESTED objects price. Why?

[Vannazux (Vanna)]

I built a bar chart graph that aggregates orders per label bucket (e.g: USD/CAD). Then I display the sum of orders price per bucket for each of the orders inside it.

My chart visualization returns a wrong price which seems to be coming approximated

This is my graph query request debug

{
  "size": 0,
  "_source": {
    "excludes": []
  },
  "aggs": {
    "3": {
      "terms": {
        "field": "label.keyword",
        "size": 2,
        "order": {
          "_term": "desc"
        }
      },
      "aggs": {
        "2": {
          "terms": {
            "field": "orders.id",
            "size": 40,
            "order": {
              "1": "desc"
            }
          },
          "aggs": {
            "1": {
              "sum": {
                "field": "orders.price"
              }
            }
          }
        }
      }
    }
  },
  "stored_fields": [
    "*"
  ],
  "script_fields": {},
  "docvalue_fields": [],
  "query": {
    "bool": {
      "must": [
        {
          "match_all": {}
        }
      ],
      "filter": [],
      "should": [],
      "must_not": []
    }
  }
}

And this is the response, which confirms the the orders.price value is being approximated

{
  "took": 9,
  "timed_out": false,
  "_shards": {
    "total": 5,
    "successful": 5,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 124,
    "max_score": 0,
    "hits": []
  },
  "aggregations": {
    "3": {
      "doc_count_error_upper_bound": 0,
      "sum_other_doc_count": 122,
      "buckets": [
        {
          "2": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "1": {
                  "value": 0.003690590019687079
                },
                "key": 213481,
                "doc_count": 1
              },
              {
                "1": {
                  "value": 0.003690590019687079
                },
                "key": 213482,
                "doc_count": 1
              },
              {
                "1": {
                  "value": 0.003690590019687079
                },
                "key": 213483,
                "doc_count": 1
              },
              {
                "1": {
                  "value": 0.003690590019687079
                },
                "key": 213484,
                "doc_count": 1
              },
              ...

And this is the photo that confirms that my data is there and should be retrieving the right price from the order.price

What is more strange is that when I flatten my data and stop using an array of nested orders, the same graph will work and display the correct value for each order.

However, I need to have this bar chart working with my source data above. My data is structured with nested orders array for each label. Am I doing anything wrong while building my graph?

[bhavyarm]

@thomasneirynck thanks

[thomasneirynck]

hi @Vannazux ,

it's hard to tell from here how much the values deviate. What are the exact values you are expecting?

I don't think we can do much on the Kibana-end for this. I do believe that Elasticsearch will approximate any kind of aggregation-values when using Terms-aggregation, and that the sorting plays into that.

Could you share some sample commands that we can copy-paste in the dev-console that illustrates the problem, so we can reproduce here?

[Vannazux (Vanna)]

Hi @thomasneirynck,

Thanks for the answer.

I am expecting the values in the blue arrows from the discovery tab picture of the orderbookindex above.

The picture shows each order has its own price. However, my query response shows that Kibana is returning the same value for every order in a wrong way. I realized that Kibana is not approximating but worst it is returning the same wrong value for every order price, look:

{
                "1": {
                  "value": 0.003690590019687079
                },
                "key": 213482,
                "doc_count": 1
              },
              {
                "1": {
                  "value": 0.003690590019687079
                },
                "key": 213483,
                "doc_count": 1
              },

I believe this has to do with some problem leading with nested types on Kibana but I am not sure.

Regarding the codes, I am new to ELK. I might not know exactly what to share, let me know what I could share that could help. Basically, I am receiving this data in orderbookindex and executing a query in visualization tab using a bar chart. The query is done by the editor displayed in the section "query debug" and the response comes as states in the query "response section" code above. I don't know how to reproduce the orderbookindex I have here but maybe if I share my index from a GET request it might help reproducing it from there.

    {
  "orderbookindex": {
    "aliases": {},
    "mappings": {
      "orderbook": {
        "properties": {
          "exchange": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "id": {
            "type": "integer"
          },
          "label": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },
          "length": {
            "type": "integer"
          },
          "orders": {
            "type": "nested",
            "properties": {
              "exchange": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "id": {
                "type": "integer"
              },
              "label": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "orderbookId": {
                "type": "integer"
              },
              "ordertype": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "pair1": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "pair2": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "price": {
                "type": "double"
              },
              "quantity": {
                "type": "double"
              },
              "timestamp": {
                "type": "text",
                "fields": {
                  "keyword": {
                    "type": "keyword",
                    "ignore_above": 256
                  }
                }
              },
              "total": {
                "type": "double"
              }
            }
          },
          "timestamp": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          }
        }
      }
    },
    "settings": {
      "index": {
        "creation_date": "1525917072851",
        "number_of_shards": "5",
        "number_of_replicas": "1",
        "uuid": "Uxj9HWRISxSKY2kRnUP4rA",
        "version": {
          "created": "6020499"
        },
        "provided_name": "orderbookindex"
      }
    }
  }
}

What call more my attention is that when I build another index with the same information of orders without using the nested "_type": "orderbook" as state in discovery tab picture, the exact Kibana query will work and retrieves the right price for each order (the prices in the blue arrows)