Using Arithmetic in pipeline.yml Processor

Elastic Stack Beats
1

Hello,
I'm appointed to update or rewriting an old ELK project. In the old version we used Logstash and its corresponding 'filebeat.cfg' file. I was rebuilding an IngestPipeline in a 'pipeline.yml'. In Losgtash we had filters and in pipeline i got processors. They are kinda similar and i was able to adjust almost everything but i've encountered one problem.

With filebeat filters we had the filter "ruby" available, in processors i don't.

This would be the ruby call from the old project:

 ruby {
            code => 'event.set("[monitor][heap][pct]", event.get("[monitor][heap][pct]").to_f * 0.01)'
            id => "MONITOR_RUBY_SCALE_HEAP_PCT"
        }

The main thing i want to solve here is this arithmetical operation. I got the field monitor.heap.pct which is mapped as float inside 'fields.yml' and now i just want to divide it by 100 resp. multiply it with 0.01.

This is a list of my available processors:

"processors": [
          {
            "type": "append"
          },
          {
            "type": "attachment"
          },
          {
            "type": "bytes"
          },
          {
            "type": "circle"
          },
          {
            "type": "community_id"
          },
          {
            "type": "convert"
          },
          {
            "type": "csv"
          },
          {
            "type": "date"
          },
          {
            "type": "date_index_name"
          },
          {
            "type": "dissect"
          },
          {
            "type": "dot_expander"
          },
          {
            "type": "drop"
          },
          {
            "type": "enrich"
          },
          {
            "type": "fail"
          },
          {
            "type": "fingerprint"
          },
          {
            "type": "foreach"
          },
          {
            "type": "geoip"
          },
          {
            "type": "grok"
          },
          {
            "type": "gsub"
          },
          {
            "type": "html_strip"
          },
          {
            "type": "inference"
          },
          {
            "type": "join"
          },
          {
            "type": "json"
          },
          {
            "type": "kv"
          },
          {
            "type": "lowercase"
          },
          {
            "type": "network_direction"
          },
          {
            "type": "pipeline"
          },
          {
            "type": "registered_domain"
          },
          {
            "type": "remove"
          },
          {
            "type": "rename"
          },
          {
            "type": "script"
          },
          {
            "type": "set"
          },
          {
            "type": "set_security_user"
          },
          {
            "type": "sort"
          },
          {
            "type": "split"
          },
          {
            "type": "trim"
          },
          {
            "type": "uppercase"
          },
          {
            "type": "uri_parts"
          },
          {
            "type": "urldecode"
          },
          {
            "type": "user_agent"
          }

Is there any way to achieve this?

Kind regards
Florian

2

"Script" would be the one to try. It will default to the painless language and should be able to do the trick.

3

This kinda solves it:

  - convert:
      if: ctx.pac != null && ctx.pac.log != null && ctx.pac.log.tags != null && ctx.pac.log.tags.contains('MONITOR')
      field: pac.log.system.monitor.cpu.pct
      type: float

  - script:
      if: ctx.pac != null && ctx.pac.log != null && ctx.pac.log.tags != null && ctx.pac.log.tags.contains('MONITOR')
      description: formatting pct for dashboards
      lang: painless
      source: ctx.pac.log.system.monitor.cpu.pct = ctx.pac.log.system.monitor.cpu.pct / 100

But why do i have to convert the fields type with the 'convert' processor at all when i configured my field as 'float' within fields.yml?
If i don't i get a script_exception caused by a runtime_error because a type conversion exception because he assumes ctx.pac.log.system.monitor.cpu.pct is a 'String'. And i assume a cast for my field would do it also, but i would like not to use any of this because it's configured as 'float'.

Kind Regards

4

Nice, glad you got it working. I'm not sure why you have to convert it to a float, but the index is logically distinct from the painless scripting language. I wouldn't worry about the resource usage of converting to the float for now.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.