Vega doesn't show/visualize any data from elasticsearch

Kibana version: 6.7.0

Elasticsearch version: 6.7.0

Original install method (e.g. download page, yum, from source, etc.): docker

Information about experience and what I am trying to achive: I do not have a lot of experience with kibana and elasticsearch(I am using the free version) I tried to explore by myself couple of weeks ago and I want to make a chart(time sires) that show information the behavior of some sensors information and mark when the behavior(error is different from 0 with other color red or blue or whatever) . I tried some examples and are listed below but they do not fetch any data I assume that the mapping(jsons) are not well mapped,fetched from elasticsearch and it doesn't recognize the fields inside.

Description of the problem including expected versus actual behavior: I am trying to plot a simple graph and it doesn't work I checked weather the query received anything(trough kibana dev tools) and I have 1000 hits but the vega is not visualizing the data in any kind of way.

The query that I am using


GET /sensor1/_search
{
  "query": {
    "range": {
      "timestamp": {
        "gte": "now-2h/d",
        "lte": "now"
      }
      
    }
  }
}

The information that I am getting from the query


{
  "took" : 1,
  "timed_out" : false,
  "_shards" : {
    "total" : 5,
    "successful" : 5,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1000,
    "max_score" : 1.0,
    "hits" : [
      {
        "_index" : "sensor1",
        "_type" : "senzors",
        "_id" : "PqBpTmoB-pKjklVPdAh5",
        "_score" : 1.0,
        "_source" : {
          "name" : "sensor1",
          "value" : 60.159078035304404,
          "timestamp" : 1556093741100,
          "error" : 2
        }
      },
      {
        "_index" : "sensor1",
        "_type" : "senzors",
        "_id" : "P6BpTmoB-pKjklVPdQgs",
        "_score" : 1.0,
        "_source" : {
          "name" : "sensor1",
          "value" : 44.50831755380234,
          "timestamp" : 1556093742100,
          "error" : 0
        }
      },
      {
        "_index" : "sensor1",
        "_type" : "senzors",
        "_id" : "JaBpTmoB-pKjklVPXAiU",
        "_score" : 1.0,
        "_source" : {
          "name" : "sensor1",
          "value" : 52.276283539084275,
          "timestamp" : 1556093716100,
          "error" : 0
        }
      },
      {
        "_index" : "sensor1",
        "_type" : "senzors",
        "_id" : "J6BpTmoB-pKjklVPXggT",
        "_score" : 1.0,
        "_source" : {
          "name" : "sensor1",
          "value" : 51.14213869391457,
          "timestamp" : 1556093718100,
          "error" : 0
        }
      },
      {
        "_index" : "sensor1",
        "_type" : "senzors",
        "_id" : "MaBpTmoB-pKjklVPZwiV",
        "_score" : 1.0,
        "_source" : {
          "name" : "sensor1",
          "value" : 51.35897156641911,
          "timestamp" : 1556093728100,
          "error" : 0
        }
      },
      {
        "_index" : "sensor1",
        "_type" : "senzors",
        "_id" : "MqBpTmoB-pKjklVPaAiE",
        "_score" : 1.0,
        "_source" : {
          "name" : "sensor1",
          "value" : 43.56251945502473,
          "timestamp" : 1556093729100,
          "error" : 0
        }
      },
      {
        "_index" : "sensor1",
        "_type" : "senzors",
        "_id" : "KaBpTmoB-pKjklVPXwi7",
        "_score" : 1.0,
        "_source" : {
          "name" : "sensor1",
          "value" : 38.130592033017045,
          "timestamp" : 1556093720100,
          "error" : 1
        }
      },
      {
        "_index" : "sensor1",
        "_type" : "senzors",
        "_id" : "IqBpTmoB-pKjklVPWQiE",
        "_score" : 1.0,
        "_source" : {
          "name" : "sensor1",
          "value" : 48.72730714815431,
          "timestamp" : 1556093713100,
          "error" : 0
        }
      },
      {
        "_index" : "sensor1",
        "_type" : "senzors",
        "_id" : "L6BpTmoB-pKjklVPZQix",
        "_score" : 1.0,
        "_source" : {
          "name" : "sensor1",
          "value" : 48.49622373903401,
          "timestamp" : 1556093726100,
          "error" : 0
        }
      },
      {
        "_index" : "sensor1",
        "_type" : "senzors",
        "_id" : "NKBpTmoB-pKjklVPagjf",
        "_score" : 1.0,
        "_source" : {
          "name" : "sensor1",
          "value" : 54.35314216737571,
          "timestamp" : 1556093731100,
          "error" : 0
        }
      }
    ]
  }
}

The code I am Trying to pass to Vega
It does not fail but it doesn't print any information


{
  "$schema": "https://vega.github.io/schema/vega/v3.json",
  "title": "Event counts from all indexes",
  "data": {
    "url": {
      "index": "sensor1",
      "body": {
        "query": {
          "range": {
            "timestamp": {
              "gte": "now-2h/d",
              "lte": "now"
            }
          }
        },
        "aggs": {
          "hist": {
            "date_histogram": {
              "field": "@timestamp",
              "interval": "3m",
              "extended_bounds": {
                "min": {"%timefilter%": "min"},
                "max": {"%timefilter%": "max"}
              },
              "min_doc_count": 0
            }
          }
        },
        "size": 0
      }
    },
    "format": {"property": "aggregations.hist.buckets"}
  },
  "mark": "line",
  "encoding": {
    "x": {
      "field": "key",
      "type": "temporal",
      "axis": {"title": "Time"}
    },
    "y": {
      "field": "error",
      "type": "temporal",
      "axis": {"title": "Document count"}
    }
  }
}

Simpler code for Vega that doesn't work eather


{
  "$schema": "https://vega.github.io/schema/vega/v3.json",
  "title": "Event counts from all indexes",
  "data": {
    "url": {
      "index": "sensor1",
      "body": {
        "query": {
          "range": {
            "timestamp": {
              "gte": "now-2h/d",
              "lte": "now"
            }
          }
        }
      }
    }
  },
  "mark": "line",
  "encoding": {
    "x": {
      "field": "key",
      "type": "temporal",
      "axis": {"title": "Time"}
    },
    "y": {
      "field": "value",
      "type": "temporal",
      "axis": {"title": "Document count"}
    }
  }
}

P.S. if someone can send me a few examples that are done with Vega or Vega-lite and Kibana and ElasticSearch (that use other fields except doc_count)

Hey @LjupchoStefanov,

Here's a working version of your query. I'll outline the changes I made:

  1. Change $schema to vega-lite v2
  2. Change date_histogram from @timestamp to timestamp to match the document properties
  3. Changed histogram interval to 5s to account for the tight grouping of data in your example dataset.
  4. Change y axis to doc_count -- I know you don't want this, but I'm working on this in steps. Now that this is rendering, I'm trying to get the errors to influence the chart. Notice I've added a sub-aggregation to collect the sum of errors in each bucket. It's returning data correctly, but I'm not using it to render anything yet.
{
  $schema: "https://vega.github.io/schema/vega-lite/v2.json",
  "title": "Event counts from all indexes",
  "data": {
    "url": {
      "index": sensor1,
      "body": {
        "query": {
          "range": {
            "timestamp": {
              "gte": "2019-04-24T08:15:11.500Z",
              "lte": "2019-04-24T08:15:44.040Z"
            }
          }
        },
        "aggs": {
          "hist": {
            "date_histogram": {
              "field": "timestamp",
              "interval": "5s", 
              "extended_bounds": {
                "min": {"%timefilter%": "min"},
                "max": {"%timefilter%": "max"}
              },
              "min_doc_count": 0
            },
            "aggs": {
        "errors": {
          "sum": {
            "field": "error"
          }
        }
      }
          }
        },
        "size": 0
      } 
    },
    "format": {"property": "aggregations.hist.buckets"}
  },
  "mark": "line",
  "encoding": {
    "x": {
      "field": "key",
      "type": "temporal",
      "axis": {"title": "Time"}
      
    },
    "y": {
      "field": "doc_count",
      "type": "quantitative",
      "axis": {"title": "Document count"}
    }
  }
}

Hey @Larry_Gregory
Thank you for your help but I actually figured it out I had a problem with the mappings and the format was not in a right order I changed the format to hist.hist and I could access all of the information (I will post a simple solution next week).

Hi

For the query mentioned below, data is not getting retrieved from elastic search.
kibana version: 6.7.0
Elasticsearch version: 6.7.0

The query for creating Visualization:
{
"$schema": "https://vega.github.io/schema/vega-lite/v2.json",
"mark": "rect",
"data": {
"url": {
"%context%": true,
"%timefield%": "@timestamp",
"index": dpk_log,
"body": {
"size": 10000,
"_source": ["child_method", "totaltime"]
}
}
"format": {"property": "hits.hits"}
}
"encoding": {
"x": {
"field": "child_method",
"type": "temporal",
"axis": {"title": "visitor name"}
},
"y": {
"field": "totaltime",
"type": "quantitative",
"axis": {"title": "Total time"}
}
}
}

in elasticsearch:
Request:
GET dpk_log/_search
{
"_source": ["child_method", "totaltime"],
"size": 2,
"query" : {
"match_all" : {
}
}
}

Response:
"hits" : {
"total" : 22,
"max_score" : 1.0,
"hits" : [
{
"_index" : "dpk_log",
"_type" : "logs",
"_id" : "j6DhU2oBIoCD8krhZ0lU",
"_score" : 1.0,
"_source" : {
"child_method" : "SourceAttributesMappingVisitor",
"totaltime" : "20"
}
},
{
"_index" : "dpk_log",
"_type" : "logs",
"_id" : "l6DhU2oBIoCD8krhZ0lZ",
"_score" : 1.0,
"_source" : {
"child_method" : "SaveDataPacketVisitor",
"totaltime" : "10"
}
}
]

Total time column in elastic search column is having null values also. Will that could be a reason for not getting visualization?

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.