"Split series" in "Line" visualization super slow

I'm splitting data in a Kibana graph (plotting multiple lines on one graph), and the splitting seems to slow everything down.

Question in short:
I'm mostly wondering if Kibana is just slow in general when splitting allot of data, or if I'm doing something wrong.

Question in long:
I'm working with results from elemental analysis, where I have two fields: "Element" and "Quantity". Element refers to chemical element (Fe, Si, Cr, etc...) and quantity is just the quantity in grams.

I want to plot this on a line graph, splitting the data on the "Element" field.

I'm able to do this by using the "Split series" bucket when creating the visualization, so I see 12 lines on the graph for 12 elements, but doing that makes the whole graph super sluggish. If I disable the "Split series" bucket, the graph shows just a single line and everything is super fast.

Here are my visualization settings:

Screenshot from 2020-06-04 22-43-17

I'd like to add that the number of hits from the Elasticsearch query are on the range 5.000 to 50.000 (depending on the source of the data and chosen interval). Does that sound like allot for Kibana to handle? Say that I'm graphing 12 lines for one year with 1 day interval, that should give me 12*365 = 4.380 dots on the graph. Is that to much?

My hunch is that I'm to blame :slight_smile: and I'm making Kibana do to much work which might be done by Elasticsearch. And that I should design the visualization and the query in some way to offload work from Kibana. Does that make sense?

More detail
To elaborate a bit then here is an example of the data that I'm working with. Part of the data (showing only 3 elements) for two measurements would look like this:

Results from midnight 2020-01-01:

  {
    "@timestamp" : "2020-01-01T00:00:00.000Z",
    "Element" : "Fe",
    "Quantity" : 21,5,
  },
  {
    "@timestamp" : "2020-01-01T00:00:00.000Z",
    "Element" : "Si",
    "Quantity" : 27.6,
  },
  {
    "@timestamp" : "2020-01-01T00:00:00.000Z",
    "Element" : "Cr",
    "Quantity" : 31.0,
  }

Results from midnight 2020-02-01:

  {
    "@timestamp" : "2020-02-01T00:00:00.000Z",
    "Element" : "Fe",
    "Quantity" : 22.3,
  },
  {
    "@timestamp" : "2020-02-01T00:00:00.000Z",
    "Element" : "Si",
    "Quantity" : 24.5,
  },
  {
    "@timestamp" : "2020-02-01T00:00:00.000Z",
    "Element" : "Cr",
    "Quantity" : 33.2,
  },

The Elasticsearch query looks like this:

{
  "aggs": {
    "2": {
      "date_histogram": {
        "field": "@timestamp",
        "calendar_interval": "1d",
        "time_zone": "Atlantic/Reykjavik",
        "min_doc_count": 1
      },
      "aggs": {
        "3": {
          "terms": {
            "field": "data.Quantifications.Element.keyword",
            "order": {
              "_key": "asc"
            },
            "size": 62
          },
          "aggs": {
            "1": {
              "avg": {
                "field": "data.Quantifications.Quantity"
              }
            }
          }
        }
      }
    }
  },
  "size": 0,
  "stored_fields": [
    "*"
  ],
  "script_1fields": {},
  "docvalue_fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    },
    {
      "field": "data.Measurement.MeasurementTimeIso",
      "format": "date_time"
    },
    {
      "field": "data.Measurement.MeasurementTimestampSeconds",
      "format": "date_time"
    }
  ],
  "_source": {
    "excludes": []
  },
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "range": {
            "@timestamp": {
              "gte": "2019-06-04T21:35:47.627Z",
              "lte": "2020-06-04T21:35:47.627Z",
              "format": "strict_date_optional_time"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

And an the first lines from the response look like this:

{
  "took": 33,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 14368,
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "2": {
      "buckets": [
        {
          "3": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "1": {
                  "value": 5.522025108337402
                },
                "key": "Cr",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 13.914688110351562
                },
                "key": "Cu",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 984.5307006835938
                },
                "key": "Fe",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 119.0231704711914
                },
                "key": "Ga",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 0.7600506544113159
                },
                "key": "Mg",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 14.317068099975586
                },
                "key": "Mn",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 79.67838287353516
                },
                "key": "Na",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 41.67288589477539
                },
                "key": "Ni",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 303.0051574707031
                },
                "key": "Si",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 37.52043533325195
                },
                "key": "Ti",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 112.84782409667969
                },
                "key": "V",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 73.44461059570312
                },
                "key": "Zn",
                "doc_count": 1
              }
            ]
          },
          "key_as_string": "2019-06-06T00:00:00.000Z",
          "key": 1559779200000,
          "doc_count": 12
        },
        {
          "3": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "1": {
                  "value": 7.688352108001709
                },
                "key": "Cr",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 15.557666778564453
                },
                "key": "Cu",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 1032.217529296875
                },
                "key": "Fe",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 119.36128997802734
                },
                "key": "Ga",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 6.439831256866455
                },
                "key": "Mg",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 14.688204765319824
                },
                "key": "Mn",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 82.25433349609375
                },
                "key": "Na",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 41.55519485473633
                },
                "key": "Ni",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 318.4305725097656
                },
                "key": "Si",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 38.49009323120117
                },
                "key": "Ti",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 113.08766174316406
                },
                "key": "V",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 72.5600814819336
                },
                "key": "Zn",
                "doc_count": 1
              }
            ]
          },
          "key_as_string": "2019-06-07T00:00:00.000Z",
          "key": 1559865600000,
          "doc_count": 12
        },
        {
          "3": {
            "doc_count_error_upper_bound": 0,
            "sum_other_doc_count": 0,
            "buckets": [
              {
                "1": {
                  "value": 7.434240341186523
                },
                "key": "Cr",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 13.125242233276367
                },
                "key": "Cu",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 963.45703125
                },
                "key": "Fe",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 120.3698959350586
                },
                "key": "Ga",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 9.350666999816895
                },
                "key": "Mg",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 14.928293228149414
                },
                "key": "Mn",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 73.5030746459961
                },
                "key": "Na",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 43.860416412353516
                },
                "key": "Ni",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 315.4002380371094
                },
                "key": "Si",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 37.861541748046875
                },
                "key": "Ti",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 111.94914245605469
                },
                "key": "V",
                "doc_count": 1
              },
              {
                "1": {
                  "value": 73.28097534179688
                },
                "key": "Zn",
                "doc_count": 1
              }
            ]
          },
          "key_as_string": "2019-06-08T00:00:00.000Z",
          "key": 1559952000000,
          "doc_count": 12
        },
        ....
        ....
        ....
        ....
        ....

Hi @dreamspy
I think the problem is related to how you order your bucket aggregations.
The way you are doing this (First aggregate by Date and then split them) means:
for every daily bucket created, I'm running a term aggregation with a max size of 100 elements.
From ES point of view, this is not a problem, but Kibana does much more computation to align all the terms/elements on every bucket before presenting them.

Reversing the aggregation order should fix this issue because you are first grouping all your elements into buckets and then for each bucket you apply a date aggregation, this should be rendered faster and in a more consistent way.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.