Create Helper Functions within Pipeline

Elastic Stack Logstash
1

I have a function in my pipeline that does some math and another that does some translation. Is there a way to create a function like in python, so that I don't have to retype the same code everytime I want to do this operation.

i.e. below, I take x,y,z coordinates and get lat,long,altitude.

    - pipeline.id: entity-state-processing
      config.string: |
        input { pipeline { address => entitystatelogs } }
        filter {
          if [attributes][entityLocation] {
            mutate {
              add_field => {
              "[location]" => "null"
              "[altitude]" => "null"
              }
            }
            # Ruby script to convert the cartesian x,y,z coordinates into lat/long
            ruby {
              init => "
                R = 6360000
                "
              code => "
                x = event.get('[attributes][entityLocation][x]').to_f
                y = event.get('[attributes][entityLocation][y]').to_f
                z = event.get('[attributes][entityLocation][z]').to_f

                range = Math.sqrt(x*x + y*y + z*z)

                lat = 180*Math.asin(z/range)/Math::PI
                long = 180*Math.atan2(y,x)/Math::PI
                alt  = range - 6369783.457722581

                event.set('[location]',  lat.to_s + ',' + long.to_s)
                event.set('[altitude]', alt.to_s)
                "
            }
          }
          if [attributes][entityType] {
            mutate {
              add_field => {
              "Entity" => "%{[attributes][entityType][entityKind]}.%{[attributes][entityType][domain]}.%{[attributes][entityType][country]}.%{[attributes][entityType][category]}.%{[attributes][entityType][subcategory]}.%{[attributes][entityType][specific]}.%{[attributes][entityType][extra]}"
              }
            }
          }
        translate {
          regex => true
          source => "Entity"
          dictionary_path => "/usr/share/logstash/advanced_mappings.json"
          }
        }
        output {
          # Sends parsed logs to elasticsearch
          elasticsearch {
            hosts => ["${OUTPUT_HOST}"]
            user => "${ELASTIC_USER}"
            password => "${ELASTIC_PASS}"
            index => "{{ .Release.Namespace }}-entity-state-%{+yyyy.MM.dd}"
            }
        }

I'm wondering, is there a way to take the ruby script and put it in a function like get_lla(x, y, z) that returns latitude, longitude, altitude.

Additionally, could I take this function:

add_field => {
              "Entity" => "%{[attributes][entityType][entityKind]}.%{[attributes][entityType][domain]}.%{[attributes][entityType][country]}.%{[attributes][entityType][category]}.%{[attributes][entityType][subcategory]}.%{[attributes][entityType][specific]}.%{[attributes][entityType][extra]}"
              }

and turn it into a function like

add_field(string first_category, string second_category, list third_categories)
2

Do you mean you want the same ruby filter in multiple pipelines, or multiple instances of very similar ruby filters in the same pipeline? If so, use a ruby script file.

3

Okay, so can I change fields from within ruby, like in the following ruby script:

def ecefToLLA(event)
    R = 6360000
    x = event.get('[attributes][entityLocation][x]').to_f
    y = event.get('[attributes][entityLocation][y]').to_f
    z = event.get('[attributes][entityLocation][z]').to_f 
    range = Math.sqrt(x*x + y*y + z*z)
    lat = 180*Math.asin(z/range)/Math::PI
    long = 180*Math.atan2(y,x)/Math::PI
    alt  = range - 6369783.457722581
    event.set('[location]',  lat.to_s + ',' + long.to_s)
    event.set('[altitude]', alt.to_s)
    return event

And in the pipeline, I would do:

filter {
    if [attributes][entityLocation] {
        mutate {
            add_field => {
                "location]" => "null"
                "[altitude]" => "null"
            }
            # Ruby script to convert the cartesian x,y,z coordinates into lat/long
            ruby {
                path => "/etc/logstash/ecefToLLA.rb"
            }
        }
    }
}
4

That should be return [event], the function has to return an array.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.