Create a propper X-Y Chart for continuous datasets (e.g. distance-force)

[mathric (Mathias)]

Hey

I'm trying to get things work in ELK. So far I created a index ('undefined') and also a pattern ('unde*'). Inside the index I am having data like this:

Measurement.Distance
Measurement.Force
Message.Timestamp
Message.DMC

If I try to create a line graph (normal one), Kibana does alwas combine Measurements of distance with digits to one number. This renders the Force-graph unusable, since I am having much more values.

As a workaround I did Measurement.DistanceX100=100*Measurement.Distance at the ingest - then I get charts I could use. But this is a bad way to fix this.

I tried with VEGA so far ... and also did a CANVAS which acktually worked quite fine -but there I cant use dashboards.

SELECT Measurement.Force as y ,Measurement.Distance as x, Message.Product.DMC AS dmc FROM "unde*"

I would need a bit of help building the VEGA code... this is how far I got ... but still receiving the INFINITY Error....

{
  "$schema": "https://vega.github.io/schema/vega-lite/v4.json",
  "title": "Event counts from all indexes",
  "data": {
    "url": {
      "%context%": true,
      "%timefield%": "Message.Timestamp",
      "index": "unde*",
      "body": {
        "aggs": {
          "time_buckets": {
            "date_histogram": {
              "field": "Message.Timestamp",
              "interval": {"%autointerval%": true},
              "extended_bounds": {
                "min": {"%timefilter%": "min"},
                "max": {"%timefilter%": "max"}
              },
              "min_doc_count": 0
            }
          }
        },
        "size": 0
      }
    },
    "format": {"property": "aggregations.time_buckets.buckets"}
  },
  "mark": "line",
  "encoding": {
    "x": {"field": "Measurement.Distance", "type": "quantitative", "axis": {"title": false}},
    "y": {
      "field": "Measurement.Force",
      "type": "quantitative",
      "axis": {"title": "Document count"}
    }
  }
}

• Infinite extent for field "Measurement.Distance": [Infinity, -Infinity]

• Infinite extent for field "Measurement.Force": [Infinity, -Infinity]

It's really not that easy to understand without more examples. I would be glad to get some hints.

Cheers Mathias

[wylie (Wylie Conlon)]

The infinite extent issue usually means that your encoding is using the wrong names for data. The typical debugging process is to use the Vega debugger to see the pipeline that Vega runs internally, and then use the right names. This is documented under our section on Vega debugging. There are also instructions there on how to ask for help with Vega in the forums, as what you've posted is not something that we can test.

[mathric (Mathias)]

Okay ... I understand the point ...

I fixed the follwing:

1. Time Selector now works (hat a issue with @timestamp ... sorry thought it would be nessesarry for the code. So replacing all with Message.Message.Timestamp
2. Getting X axis to distance parameter and doing buckets every 0.05mm works

{
  "$schema": "https://vega.github.io/schema/vega-lite/v4.json",
  "title": "Event counts from all indexes",
  "data": {
    "url": {
      "%context%": true,
      "%timefield%": "Message.Message.Timestamp",
      "index": "unde*",
      "body": {
        "aggs": {
          "distance": {
            "histogram": {
              "field": "Message.Measurement.Dataset.Distance",
              "interval": 0.05
            }
          }
        },
        "size": 0
      }
    },
    "format": {"property": "aggregations.distance.buckets"}
  },
  "mark": "line",
  "encoding": {
    "x": {
      "field": "key",
      "type": "quantitative",
      "axis": {"title": "Distance"}
    },
    "y": {
      "field": "doc_count",
      "type": "quantitative",
      "axis": {"title": "Force"}
    }
  }
}

I'm getting the data from this request like this (shortened a bit) by Vega-Inspect RESPONSE:

{
  "took": 7,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": 2196,
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "distance": {
      "buckets": [
        {
          "key": 154.95000000000002,
          "doc_count": 10
        },
        {
          "key": 155,
          "doc_count": 0
        },
        {
          "key": 155.05,
          "doc_count": 0
        }
      ]
    }
  }
}

looking to the Vega Debug-SPEC:

{
  "$schema": "https://vega.github.io/schema/vega-lite/v4.json",
  "title": "Event counts from all indexes",
  "data": {
    "format": {
      "property": "aggregations.distance.buckets"
    },
    "values": {
      "took": 7,
      "timed_out": false,
      "_shards": {
        "total": 1,
        "successful": 1,
        "skipped": 0,
        "failed": 0
      },
      "hits": {
        "total": 2196,
        "max_score": null,
        "hits": []
      },
      "aggregations": {
        "distance": {
          "buckets": [
            {
              "key": 154.95000000000002,
              "doc_count": 10
            },
            {
              "key": 155,
              "doc_count": 0
            },
            {
              "key": 155.05,
              "doc_count": 0
            },
            {
              "key": 155.10000000000002,
              "doc_count": 0
            },
            {
              "key": 286.1,
              "doc_count": 12
            }
          ]
        }
      }
    }
  },
  "mark": "line",
  "encoding": {
    "x": {
      "field": "key",
      "type": "quantitative",
      "axis": {
        "title": "Distance"
      }
    },
    "y": {
      "field": "doc_count",
      "type": "quantitative",
      "axis": {
        "title": "Force"
      }
    }
  },
  "config": {
    "range": {
      "category": {
        "scheme": "elastic"
      }
    },
    "mark": {
      "color": "#54B399"
    },
    "title": {
      "color": "#343741"
    },
    "style": {
      "guide-label": {
        "fill": "#69707d"
      },
      "guide-title": {
        "fill": "#343741"
      },
      "group-title": {
        "fill": "#343741"
      },
      "group-subtitle": {
        "fill": "#343741"
      }
    },
    "axis": {
      "tickColor": "#eef0f3",
      "domainColor": "#eef0f3",
      "gridColor": "#eef0f3"
    },
    "background": "transparent"
  },
  "width": "container",
  "height": "container",
  "autosize": {
    "type": "fit",
    "contains": "padding"
  }
}

Next would be to exchange the doc_count with a average value of the term Message.Measurement.Dataset.Force
and a Sub Agregation to the ID of the dataset: Message.Product.DMC

But I don't find the correct way to put the request - As far as I understand, I need to add the data to the request to have it available in the chart.

I tried this but got "500 Bad Gateway":

...
      "body": {
        "aggs": {
          "distance": {
            "dmc":{"field:":"Message.Product.DMC.keyword"},
            "force":{"field:":"Message.Measurement.Dataset.Force"},
            "histogram": {
              "field": "Message.Measurement.Dataset.Distance",
              "interval": 0.05
            }
          }
...

In the Index the Value Message.Product.DMC.keyword (and of course Message.Product.DMC) exists. I use this Value for normal Kibana Charts too (like the others).

Maybe I missunderstand the function of the body / aggs section.

[wylie (Wylie Conlon)]

500 Bad gateway is usually a connection issue, like in a proxy. You may need to retry your request.