Need help on arithmetic operators usage in ingestion / scripted fields

[prabhakar_talari]

Hi All,

I have a field called duration which is number ( in seconds ). I would like to write ingestion pipeline OR scripted field add new field called duration_hr by dividing the duration with 3600. my ES version is 7.6 also i could see in scripted field only painless language is supported same for ingestion pipeline as well.

Also, i do not want to use any Kibana visualization for this i am primarily focusing on how i can add new field to document which is duraiton_hr.

Could some one suggest how could i achieve this

Thanks

[ylasri (Yassine LASRI)]

Note for scripted fields

Computing data on the fly with scripted fields can be very resource intensive and can have a direct impact on Kibana performance. Keep in mind that there’s no built-in validation of a scripted field. If your scripts are buggy, you’ll get exceptions whenever you try to view the dynamically generated data.

This can be achiedved like this :

Example of index

PUT my_index
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0
  },
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "call_duration_ms": {
        "type": "long"
      },
       "call_duration_h": {
        "type": "long"
      }
    }
  }
}

Example of document ingested

POST my_index/_doc
{
  "@timestamp": "2020-09-06T09:48:10.000",
  "call_duration_ms": 7200
}

This is an example of search query that add a scripted field on the flight during the search (the scripted field can be added in kibana on the index pattern used with the code : doc['call_duration_ms'].value / 3600.00

GET my_index/_search
{
  "_source": [
    "@timestamp",
    "call_duration"
  ],
  "script_fields": {
    "duration_h": {
      "script": {
        "source": "doc['call_duration_ms'].value / params.factor",
        "params": {
          "factor": 3600.00
        }
      }
    }
  }
}

I would suggest an ingest pipeline as it intercept documents and compute the field then ingest ...

PUT _ingest/pipeline/my_index
{
  "description": "this is an example to get duration in hour",
  "processors": [
    {
      "script": {
        "lang": "painless",
        "source": "ctx.call_duration_h  = ctx.call_duration_ms  / params.factor",
        "params": {
          "factor": 3600.00
        }
      }
    }
  ]
}

The ingest pipeline can be trigged when ingesting documents like this :

POST my_index/_doc?pipeline=my_index
{
  "@timestamp": "2020-09-06T10:48:10.000",
  "call_duration": 9200
}

Or it can setted up in the settings of your index

PUT my_index
{
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 0,
    "index.default_pipeline": "my_index",
    "index.final_pipeline": "foo"

  },
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "call_duration_ms": {
        "type": "long"
      },
       "call_duration_h": {
        "type": "long"
      }
    }
  }
}

If you are using logstash as ingest tool, i prefer using a ruby filter to that calculation

[prabhakar_talari]

Thank you so much.. i have implemented ingestion pipeline things looks good.