How To Improve Kibana Performance

Elastic Stack Kibana
1

Hi,

I have a index that contains almost 8 million data. I wanna draw a graph with vega lite and it should contains almost 800000 data, but when i try this with vega visualization, kibana is not responding and it shuts down elasticsearch and itself. Can i improve kibana performance in a way?

Other question, is kibana using gpu while rendering a graph? or if it is not using, is there a settings to make it ?

2

Hi @Sinan_Kaplan,

thanks for reaching out.

Can you share your vega spec? This makes it easier to help with your specific case.

In general, there are a lot of different bottlenecks you can hit when creating a visualization - Elasticsearch, amount of data to download to the client, the capabilities of your end device to actually render it, ...
Vega is using canvas to render the actualy visualization, it's currently not possible to change this.

If you are trying to visualize 800k data points in a single visualization you have to ask yourself whether you as a human can even make sense out of this amount of data - it might make more sense to aggregate or sample the data down somehow to not get overwhelmed.

3 
{
  "$schema": "https://vega.github.io/schema/vega-lite/v2.4.0.json",
  "data": {
    "url": {
      "index": "n_36bca9821e",
      "body": {
        "query": {
          "bool": {
            "must": [
              {
                "bool": {
                  "should": [
                    {
                      "match": {
                        "data.type": 1
                      }
                    },
                    {
                      "match": {
                        "data.type": 4
                      }
                    }
                  ]
                }
              }
            ]
          }
        },
        "size": 800000
      }
    },
    "format": {
      "property": "hits.hits"
    }
  },
  "transform": [
    {
      "calculate": "toDate(datum._source['timestamp'])",
      "as": "time"
    },
    {
      "calculate": "datum._source.data.type == '1' ? 'Type1' : 'Other'",
      "as": "dataType"
    },
    {
      "calculate": "datum._source.data.sub_type",
      "as": "subType"
    },
    {
      "calculate": "  datum._source.data.type=='4'&& subTypedatum._source.data.detail.resType == '1' ? 'Type1 Fail' : 'AllFail' ",
      "as": "Fail"
    }
  ],
  "layer": [
    {
      "transform": [
        {
          "filter": {
            "field": "dataType",
            "oneOf": [
              "Type1"
            ]
          }
        }
      ],
      "mark": {
        "opacity": 0.7,
        "type": "bar"
      },
      "encoding": {
        "x": {
          "field": "dataType",
          "type": "nominal",
          "axis": {
            "title": "Types"
          }
        },
        "y": {
          "aggregate": "count",
          "field": "dataType",
          "type": "quantitative",
          "axis": {
            "title": "Count",
            "titleColor": "#85C5A6"
          }
        },
        "color": {
          "field": "subType",
          "type": "nominal",
          "legend": null
        }
      }
    },
    {
      "transform": [
        {
          "filter": {
            "field": "Fail",
            "oneOf": [
              "Type1 Fail"
            ]
          }
        }
      ],
      "mark": {
        "opacity": 0.5,
        "type": "bar",
        "color": "#ECFF00"
      },
      "encoding": {
        "x": {
          "field": "Fail",
          "type": "nominal",
          "axis": {
            "title": "Types",
            "labelAngle": 0
          }
        },
        "y": {
          "aggregate": "count",
          "field": "Fail",
          "type": "quantitative",
          "axis": {
            "title": "Count"
          }
        },
        "color": {
          "field": "subType",
          "type": "nominal",
          "legend": {
            "title": "SubTypes"
          }
        }
      }
    }
  ]
}

Hi, @flash1293,
Thanks for your reply.
I have a graph like this and i dont want to show a point graph. I just wanna show 2 different count in same graph like bar graph.

4

Hi, it looks like you could achieve this with aggregations instead of working on the raw documents and doing the aggregation client-side. If you move this to the server, it will work much faster because you don't have to move that much data around.

Try to do a terms aggregation by data.type and sub_type and a filters aggregation to split out the "Type 1 Fail" cases and work with the resulting counts directly instead of using vega aggregates.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.