Calculating new fields using ruby filter

Elastic Stack Logstash
1

Hi evryone
i tried o use ruby filter to calculate 3 new fields:
The first is the total number of observation in may table.
the second id the number of observations where(Contrainte>=-78),Contrainte is a field in my csv file.
The third is the percentage of compliant observations.
Unfortuntally it doesn't work.
This is my logstash config file:

input{

    file{

      path => "C:/Users/Asus/Dropbox/PFE_part2/moov_dogbo_2g.csv"

        start_position => "beginning"

        sincedb_path => "NUL"

        codec => plain { charset => "CP1252" }

    }

}

filter{

    csv {

        separator => ","

        columns => [ "Message",

                     "Time",

                     "Distance",

                     "Longitude",

                     "Latitude",

                     "ServRxLevIdle"

                    ]

        convert => {

            "Longitude" => "float"

            "Latitude" => "float"

            "ServRxLevIdle" => "float" 

        } 

    }

    mutate { rename => ["ServRxLevIdle",  Contrainte] }

    mutate { add_field => { "Location" => ["%{[Latitude]}","%{[Longitude]}"] } }   

    mutate { convert=> ["Location",  "float"] }

    date { match => [ "time", "dd MMM yy HH:mm:ss" ] }

    ruby {  code => 'event.set("[total_observation_number]", event.get("Message").count())'} #total number of observation

    if(["Contrainte"] >= -78){

        ruby { code =>'event.set("[observation_compliant]", event.get("Message").count())'}  #number of compliant observation

    }

    ruby{ code => 'event.set("[percentage]", [event.get("[observation_compliant]") / event.get("[total_observation_number]") ] * 100  )'}

}

output{

        elasticsearch { 

                    action => "index"

                    hosts => ["http://localhost:9200/"] 

                    index => "friends"

                   

  }

        stdout { codec => rubydebug }

}

And here is part of the error

        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
warning: thread "[main]>worker2" terminated with exception (report_on_exception is true):
java.lang.IllegalStateException: java.lang.NullPointerException
        at org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:85)
        at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:441)
        at org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:305)
        at C_3a_.elastic_stack.logstash_minus_7_dot_6_dot_2.logstash_minus_core.lib.logstash.java_pipeline.start_workers(C:/elastic_stack/logstash-7.6.2/logstash-core/lib/logstash/java_pipeline.rb:262)
        at org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)
        at java.lang.Thread.run(java/lang/Thread.java:748)
Caused by: java.lang.NullPointerException
        at org.logstash.config.ir.compiler.EventCondition$Compiler$UnexpectedTypeException.<init>(EventCondition.java:659)
        at org.logstash.config.ir.compiler.EventCondition$Compiler.compare(EventCondition.java:433)
        at org.logstash.config.ir.compiler.EventCondition$Compiler.lambda$compareFieldToConstant$11(EventCondition.java:424)
        at org.logstash.config.ir.compiler.Utils.filterEvents(Utils.java:27)
        at org.logstash.generated.CompiledDataset6.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset7.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset8.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset9.compute(Unknown Source)
        at org.logstash.generated.CompiledDataset10.compute(Unknown Source)
        at org.logstash.execution.WorkerLoop.run(WorkerLoop.java:64)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:441)
        at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:305)
        at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:32)
        at C_3a_.elastic_stack.logstash_minus_7_dot_6_dot_2.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_workers$5(C:/elastic_stack/logstash-7.6.2/logstash-core/lib/logstash/java_pipeline.rb:262)
        at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136)
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77)
        at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)
        at org.jruby.runtime.Block.call(Block.java:125)
        at org.jruby.RubyProc.call(RubyProc.java:274)
        at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)
        at java.lang.Thread.run(Thread.java:748)
[2020-06-15T17:31:48,834][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>java.lang.IllegalStateException: java.lang.NullPointerException, :backtrace=>["org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:85)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:441)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:305)", "C_3a_.elastic_stack.logstash_minus_7_dot_6_dot_2.logstash_minus_core.lib.logstash.java_pipeline.start_workers(C:/elastic_stack/logstash-7.6.2/logstash-core/lib/logstash/java_pipeline.rb:262)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "java.lang.Thread.run(java/lang/Thread.java:748)"]}
[2020-06-15T17:31:48,858][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

Can you help me?

2

That is comparing an array of strings with a number. logstash does not handle that well.

Note that if you fix it to compare the field called Contrainte

if [Contrainte] >= -78 {

then you will still get an exception if the field does not exist.

3

Thank's for help, but even when i change it, it still dosen't work .
i didn't found the index in elasticsearch.
This is what i get:

Sending Logstash logs to C:/ELK/logstash/logstash-7.6.2/logs which is now configured via log4j2.properties
[2020-06-15T22:06:15,064][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-06-15T22:06:15,194][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.6.2"}
[2020-06-15T22:06:19,237][INFO ][org.reflections.Reflections] Reflections took 48 ms to scan 1 urls, producing 20 keys and 40 values
[2020-06-15T22:06:24,669][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2020-06-15T22:06:25,987][WARN ][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2020-06-15T22:06:26,195][INFO ][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}
[2020-06-15T22:06:26,201][WARN ][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2020-06-15T22:06:26,328][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200/"]}
[2020-06-15T22:06:26,404][INFO ][logstash.outputs.elasticsearch][main] Using default mapping template
[2020-06-15T22:06:26,603][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2020-06-15T22:06:26,666][WARN ][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been created for key: cluster_uuids. This may result in invalid serialization.  It is recommended to log an issue to the responsible developer/development team.
[2020-06-15T22:06:26,738][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["C:/Users/SOUMAYA/Desktop/newlogstash.conf"], :thread=>"#<Thread:0x7ab7735c run>"}
[2020-06-15T22:06:29,713][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2020-06-15T22:06:29,809][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-06-15T22:06:29,810][INFO ][filewatch.observingtail  ][main] START, creating Discoverer, Watch with file and sincedb collections
[2020-06-15T22:06:30,454][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

Could you help me?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.