Run time fields in Kibana VIsualizations

[JeremyP]

Hello,

I have a couple of run time fields defined in the index mappings which calculates the difference between two time stamps in days. It works fine, and I can see the data in Kibana discover. However, if I attempt to query the field using KQL, it shows no results (either using gt or lt expressions or simple existence). Both are mapped to "long" fields. I cannot use them in visualizations. I can see them in my queries in dev tools.

I have other run time fields which are keyword based, and they work just fine. Is there something I'm missing? Should this work?

Thank you.

[Priscilla_Parodi]

Hello @JeremyP,

What version are you using? Could you please share with me the KQL query you're using that is not showing results? Additionally, could you also share the query that is showing results?

Note: You can also use a Query DSL filter.

[JeremyP]

Hi,

I'm running 8.9. Here is the run time field I applied to the index.

put dev-xvulnerability-18-2023.09.27/_mapping
{
  "properties": {
    "kpi.vulnerability_age_days": {
      "type": "long",
      "script": {
        "source": """
          if (doc["vulnerability.state.keyword"].value == "remediated") { 
            emit((doc["vulnerability.last_assessed_for_vulnerabilities"].value.millis - doc["nexpos.vulnerability.instance.date_found"].value.millis) / 1000 / 60 / 60 / 24) }
          else {
            emit((new Date().getTime() - doc["nexpose.vulnerability.instance.date_found"].value.millis) / 1000 / 60 / 60 / 24) }
        """,
        "lang": "painless"
      }
    }
  }
}

Screenshot of the KPI field and no field data...

KQL query....

kpi.vulnerability_age_days : * 

This comes back with no results.

Here is a DSL query....

GET dev-xvulnerability-18-2023.09.27/_search
{
  "_source": [
    "kpi.vulnerability_age_days"], 
  "fields": [
    "kpi.vulnerability_age_days"
  ]
}

Sample results....

{
  "took": 4,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 2747,
      "relation": "eq"
    },
    "max_score": 1,
    "hits": [
      {
        "_index": "dev-xvulnerability-18-2023.09.27",
        "_id": "18-win2016-1-CVE-2023-29351",
        "_score": 1,
        "_source": {},
        "fields": {
          "kpi.vulnerability_age_days": [
            111
          ]
        }
      },
      {
        "_index": "dev-xvulnerability-18-2023.09.27",
        "_id": "18-win2016-1-CVE-2023-29372",
        "_score": 1,
        "_source": {},
        "fields": {
          "kpi.vulnerability_age_days": [
            111
          ]
        }
      },
      {
        "_index": "dev-xvulnerability-18-2023.09.27",
        "_id": "18-win2016-1-CVE-2023-37205",
        "_score": 1,
        "_source": {},
        "fields": {
          "kpi.vulnerability_age_days": [
            87
          ]
        }
      },

If I modify my component template to put in the run time field and re-index the data, the data is usable in Kibana. It shows up in the visualizations and I can use KQL. From my limited experience with runtime fields, I was under the impression that re-indexing is not required and those fields are calculated at search.

Thank you for responding!

[JeremyP]

Hi @Priscilla_Parodi.... I forgot to tag you in my reply. See above for the details you requested. Thank you for taking the time.

[Priscilla_Parodi]

@JeremyP

From my limited experience with runtime fields, I was under the impression that re-indexing is not required and those fields are calculated at search.

Correct. They are defined in the index mapping or in the query, and once defined they are immediately available for search requests, aggregations, filtering, and sorting.

Could you see this field in the Index Pattern?

[JeremyP]

Hi @Priscilla_Parodi,

If you are referring to the data view, yes, the field is present. See screenshot.

Is it possible we have a bug? I do have an elastic support case opened on this as well but it's in the early phases of review.

Thank you!

[Priscilla_Parodi]

Could you please filter by runtime?

[Priscilla_Parodi]

Btw pls check the mapping of the runtime field. This is a good tutorial. Map a runtime field.

[JeremyP]

I get no items found.

[JeremyP]

@Priscilla_Parodi I did not have it mapped as a run-time. When I re-indexed, it created the runtime field as I've explicitly defined so in the component template. Once I determined this, I added the right mapping.

Thanks for leading me to this conclusion. Much appreciated.