Using Arithmetic in pipeline.yml Processor

Hello,
I'm appointed to update or rewriting an old ELK project. In the old version we used Logstash and its corresponding 'filebeat.cfg' file. I was rebuilding an IngestPipeline in a 'pipeline.yml'. In Losgtash we had filters and in pipeline i got processors. They are kinda similar and i was able to adjust almost everything but i've encountered one problem.

With filebeat filters we had the filter "ruby" available, in processors i don't.

This would be the ruby call from the old project:

 ruby {
            code => 'event.set("[monitor][heap][pct]", event.get("[monitor][heap][pct]").to_f * 0.01)'
            id => "MONITOR_RUBY_SCALE_HEAP_PCT"
        }

The main thing i want to solve here is this arithmetical operation. I got the field monitor.heap.pct which is mapped as float inside 'fields.yml' and now i just want to divide it by 100 resp. multiply it with 0.01.

This is a list of my available processors:

"processors": [
          {
            "type": "append"
          },
          {
            "type": "attachment"
          },
          {
            "type": "bytes"
          },
          {
            "type": "circle"
          },
          {
            "type": "community_id"
          },
          {
            "type": "convert"
          },
          {
            "type": "csv"
          },
          {
            "type": "date"
          },
          {
            "type": "date_index_name"
          },
          {
            "type": "dissect"
          },
          {
            "type": "dot_expander"
          },
          {
            "type": "drop"
          },
          {
            "type": "enrich"
          },
          {
            "type": "fail"
          },
          {
            "type": "fingerprint"
          },
          {
            "type": "foreach"
          },
          {
            "type": "geoip"
          },
          {
            "type": "grok"
          },
          {
            "type": "gsub"
          },
          {
            "type": "html_strip"
          },
          {
            "type": "inference"
          },
          {
            "type": "join"
          },
          {
            "type": "json"
          },
          {
            "type": "kv"
          },
          {
            "type": "lowercase"
          },
          {
            "type": "network_direction"
          },
          {
            "type": "pipeline"
          },
          {
            "type": "registered_domain"
          },
          {
            "type": "remove"
          },
          {
            "type": "rename"
          },
          {
            "type": "script"
          },
          {
            "type": "set"
          },
          {
            "type": "set_security_user"
          },
          {
            "type": "sort"
          },
          {
            "type": "split"
          },
          {
            "type": "trim"
          },
          {
            "type": "uppercase"
          },
          {
            "type": "uri_parts"
          },
          {
            "type": "urldecode"
          },
          {
            "type": "user_agent"
          }

Is there any way to achieve this?

Kind regards
Florian

"Script" would be the one to try. It will default to the painless language and should be able to do the trick.

This kinda solves it:

  - convert:
      if: ctx.pac != null && ctx.pac.log != null && ctx.pac.log.tags != null && ctx.pac.log.tags.contains('MONITOR')
      field: pac.log.system.monitor.cpu.pct
      type: float

  - script:
      if: ctx.pac != null && ctx.pac.log != null && ctx.pac.log.tags != null && ctx.pac.log.tags.contains('MONITOR')
      description: formatting pct for dashboards
      lang: painless
      source: ctx.pac.log.system.monitor.cpu.pct = ctx.pac.log.system.monitor.cpu.pct / 100

But why do i have to convert the fields type with the 'convert' processor at all when i configured my field as 'float' within fields.yml?
If i don't i get a script_exception caused by a runtime_error because a type conversion exception because he assumes ctx.pac.log.system.monitor.cpu.pct is a 'String'. And i assume a cast for my field would do it also, but i would like not to use any of this because it's configured as 'float'.

Kind Regards

Nice, glad you got it working. I'm not sure why you have to convert it to a float, but the index is logically distinct from the painless scripting language. I wouldn't worry about the resource usage of converting to the float for now.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.