Vega request @timestamp not mapped to timestamp field

Elastic Stack Kibana
1

Hi,

I am using Kibana and ES 7.10.0.

I have a vega-lite widget that works and fetches from Elasticsearch the data. When trying using @timestamp in a vega-lite widget I get no results, and the warning: "Infinite extent for field "doc_count": [Infinity, -Infinity]".

Ref: https://www.elastic.co/guide/en/kibana/current/vega-reference.html#vega-queries

{
  "$schema": "https://vega.github.io/schema/vega-lite/v4.json",
  "data": {
    "url": {
      // Filter the time picker (upper right corner) with this field
      %timefield%: @timestamp
      // Apply dashboard context filters when set
      %context%: true
      "index": "my_index",
      "body": {
        "aggs": {
          "aaa": {
            "nested": {"path": "xxx"},
            "aggs": {
              "bbb": {
                "nested": {"path": "xxx.yyy"},
                "aggs": {
                  "ccc": {
                    "terms": {
                      "field": "xxx.yyy.zzz",
                      "size": 4
                    }
                  }
                }
              }
            }
          }
        },
        "size": 0
      }
    },
    "format": {"property": "aggregations.aaa.bbb.ccc.buckets"}
  },
  "encoding": {
    "x": {"field": "key", "type": "nominal", "axis": {"title": false}},
    "y": {
      "field": "doc_count",
      "type": "quantitative",
      "axis": {"title": "Document count"}
    }
  },
  "layer": [
    {"mark": "bar"},
    {
      "mark": {"type": "text", "align": "left", "baseline": "middle", "dy": 10},
      "encoding": {"text": {"field": "doc_count", "type": "quantitative"}}
    }
  ]
}

Using the Inspect tool I can see that the request to ES contains the following:

// rest of code left out for clarity
"range": {
  "@timestamp": {
    "gte": "2018-12-01T11:37:26.326Z",
    "lte": "2020-12-01T11:37:26.326Z",
    "format": "strict_date_optional_time"
  }
}

Shouldn't @timestamp have been replaced with the Index Pattern's @timestamp ('creation_timestamp' in my case).

When I remove '%timefield%: @timestamp' and explicitly add a query clause with the timestamp as follows, it works well:

// rest of code left out for clarity
"range": {
  "creation_timestamp": {
    "%timefilter%": true,
    "format": "strict_date_optional_time"
   }
}

I double-checked that my Index Pattern has @timestamp set to 'creation_timestamp'.

What am I doing wrong?

2

Have you tried %timefield%: creation_timestamp

3

Adding %timefield%: creation_timestamp works, but the question remains. Shouldn't it work with @timestamp?

4

I am not sure how it was setup or what your data/mapping looks like so I am not sure if that method works or not, never tried it.

But I would just use the field you want to work with and it seems like it's creation_timestamp unless I am missing something.

5

You need to specify the exact field name, Vega isn't able to guess.

6

Thanks, understood.
Looking back there are probably 2-3 things that led me to think that @timestamp is a 'feature':

  1. The Kibana vega template uses @timestamp. And results for _all are fetched.
  2. The documentation uses @timestamp and has a note which reads: " @timestamp — Filters the time range and breaks it into histogram buckets."
  3. When you create an Index Pattern you get to specify the timestamp for the index.

But now I realize that the Kibana vega template is using @timestamp and fetches results only from .kibana-event-log-7.10.0-000001 log (on my setup), which has a property of @timestamp and that is the only reason why this had worked, and has nothing to do with the Index Pattern as the vega specifies the index directly.

I would suggest changing the comment in the documentation from " @timestamp — Filters the time range and breaks it into histogram buckets." so that it will be clear that you need to use an actual property name from your index.

Thanks.

7

Thanks for the suggestion, I think it makes sense to clarify this. The field name @timestamp is standardized as part of our Elastic Common Schema, but this isn't always obvious.

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.