# Calculating new fields using ruby filter
**URL:** https://discuss.elastic.co/t/calculating-new-fields-using-ruby-filter/237163
**Category:** Logstash
**Created:** [June 15, 2020, 3:49pm UTC](https://discuss.elastic.co/t/calculating-new-fields-using-ruby-filter/237163 "2020-06-15T15:49:46Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@inchirah](https://discuss.elastic.co/u/inchirah)
#### Post date: [June 15, 2020, 3:49pm UTC](https://discuss.elastic.co/t/calculating-new-fields-using-ruby-filter/237163/1 "2020-06-15T15:49:47Z")
Hi evryone
i tried o use ruby filter to calculate 3 new fields:
The first is the total number of observation in may table.
the second id the number of observations where(Contrainte\>=-78),Contrainte is a field in my csv file.
The third is the percentage of compliant observations.
Unfortuntally it doesn't work.
This is my logstash config file:
```auto
input{
file{
path => "C:/Users/Asus/Dropbox/PFE_part2/moov_dogbo_2g.csv"
start_position => "beginning"
sincedb_path => "NUL"
codec => plain { charset => "CP1252" }
}
}
filter{
csv {
separator => ","
columns => [ "Message",
"Time",
"Distance",
"Longitude",
"Latitude",
"ServRxLevIdle"
]
convert => {
"Longitude" => "float"
"Latitude" => "float"
"ServRxLevIdle" => "float"
}
}
mutate { rename => ["ServRxLevIdle", Contrainte] }
mutate { add_field => { "Location" => ["%{[Latitude]}","%{[Longitude]}"] } }
mutate { convert=> ["Location", "float"] }
date { match => ["time", "dd MMM yy HH:mm:ss"] }
ruby { code => 'event.set("[total_observation_number]", event.get("Message").count())'} #total number of observation
if(["Contrainte"] >= -78){
ruby { code =>'event.set("[observation_compliant]", event.get("Message").count())'} #number of compliant observation
}
ruby{ code => 'event.set("[percentage]", [event.get("[observation_compliant]") / event.get("[total_observation_number]") ] * 100 )'}
}
output{
elasticsearch {
action => "index"
hosts => ["http://localhost:9200/"]
index => "friends"
}
stdout { codec => rubydebug }
}
```
And here is part of the error
```auto
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
warning: thread "[main]>worker2" terminated with exception (report_on_exception is true):
java.lang.IllegalStateException: java.lang.NullPointerException
at org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:85)
at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)
at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:441)
at org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:305)
at C_3a_.elastic_stack.logstash_minus_7_dot_6_dot_2.logstash_minus_core.lib.logstash.java_pipeline.start_workers(C:/elastic_stack/logstash-7.6.2/logstash-core/lib/logstash/java_pipeline.rb:262)
at org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)
at java.lang.Thread.run(java/lang/Thread.java:748)
Caused by: java.lang.NullPointerException
at org.logstash.config.ir.compiler.EventCondition$Compiler$UnexpectedTypeException.(EventCondition.java:659)
at org.logstash.config.ir.compiler.EventCondition$Compiler.compare(EventCondition.java:433)
at org.logstash.config.ir.compiler.EventCondition$Compiler.lambda$compareFieldToConstant$11(EventCondition.java:424)
at org.logstash.config.ir.compiler.Utils.filterEvents(Utils.java:27)
at org.logstash.generated.CompiledDataset6.compute(Unknown Source)
at org.logstash.generated.CompiledDataset7.compute(Unknown Source)
at org.logstash.generated.CompiledDataset8.compute(Unknown Source)
at org.logstash.generated.CompiledDataset9.compute(Unknown Source)
at org.logstash.generated.CompiledDataset10.compute(Unknown Source)
at org.logstash.execution.WorkerLoop.run(WorkerLoop.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(JavaMethod.java:441)
at org.jruby.javasupport.JavaMethod.invokeDirect(JavaMethod.java:305)
at org.jruby.java.invokers.InstanceMethodInvoker.call(InstanceMethodInvoker.java:32)
at C_3a_.elastic_stack.logstash_minus_7_dot_6_dot_2.logstash_minus_core.lib.logstash.java_pipeline.RUBY$block$start_workers$5(C:/elastic_stack/logstash-7.6.2/logstash-core/lib/logstash/java_pipeline.rb:262)
at org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136)
at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77)
at org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:71)
at org.jruby.runtime.Block.call(Block.java:125)
at org.jruby.RubyProc.call(RubyProc.java:274)
at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)
at java.lang.Thread.run(Thread.java:748)
[2020-06-15T17:31:48,834][FATAL][logstash.runner] An unexpected error occurred! {:error=>java.lang.IllegalStateException: java.lang.NullPointerException, :backtrace=>["org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:85)", "java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498)", "org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:441)", "org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:305)", "C_3a_.elastic_stack.logstash_minus_7_dot_6_dot_2.logstash_minus_core.lib.logstash.java_pipeline.start_workers(C:/elastic_stack/logstash-7.6.2/logstash-core/lib/logstash/java_pipeline.rb:262)", "org.jruby.RubyProc.call(org/jruby/RubyProc.java:274)", "java.lang.Thread.run(java/lang/Thread.java:748)"]}
[2020-06-15T17:31:48,858][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
```
Can you help me?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [June 15, 2020, 5:18pm UTC](https://discuss.elastic.co/t/calculating-new-fields-using-ruby-filter/237163/2 "2020-06-15T17:18:03Z")
> [@inchirah](#):
>
> `if(["Contrainte"] >= -78){`
That is comparing an array of strings with a number. logstash does not handle that well.
Note that if you fix it to compare the field called Contrainte
```
if [Contrainte] >= -78 {
```
then you will [still](https://discuss.elastic.co/t/comparing-number-in-conditionnal-event-treatement/217043/5) get an exception if the field does not exist.
---
### Author:  [@inchirah](https://discuss.elastic.co/u/inchirah)
#### Post date: [June 15, 2020, 9:13pm UTC](https://discuss.elastic.co/t/calculating-new-fields-using-ruby-filter/237163/3 "2020-06-15T21:13:28Z")
Thank's for help, but even when i change it, it still dosen't work .
i didn't found the index in elasticsearch.
This is what i get:
```auto
Sending Logstash logs to C:/ELK/logstash/logstash-7.6.2/logs which is now configured via log4j2.properties
[2020-06-15T22:06:15,064][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2020-06-15T22:06:15,194][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"7.6.2"}
[2020-06-15T22:06:19,237][INFO][org.reflections.Reflections] Reflections took 48 ms to scan 1 urls, producing 20 keys and 40 values
[2020-06-15T22:06:24,669][INFO][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2020-06-15T22:06:25,987][WARN][logstash.outputs.elasticsearch][main] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2020-06-15T22:06:26,195][INFO][logstash.outputs.elasticsearch][main] ES Output version determined {:es_version=>7}
[2020-06-15T22:06:26,201][WARN][logstash.outputs.elasticsearch][main] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[2020-06-15T22:06:26,328][INFO][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://localhost:9200/"]}
[2020-06-15T22:06:26,404][INFO][logstash.outputs.elasticsearch][main] Using default mapping template
[2020-06-15T22:06:26,603][INFO][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
[2020-06-15T22:06:26,666][WARN][org.logstash.instrument.metrics.gauge.LazyDelegatingGauge][main] A gauge metric of an unknown type (org.jruby.specialized.RubyArrayOneObject) has been created for key: cluster_uuids. This may result in invalid serialization. It is recommended to log an issue to the responsible developer/development team.
[2020-06-15T22:06:26,738][INFO][logstash.javapipeline][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["C:/Users/SOUMAYA/Desktop/newlogstash.conf"], :thread=>"#"}
[2020-06-15T22:06:29,713][INFO][logstash.javapipeline][main] Pipeline started {"pipeline.id"=>"main"}
[2020-06-15T22:06:29,809][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2020-06-15T22:06:29,810][INFO][filewatch.observingtail][main] START, creating Discoverer, Watch with file and sincedb collections
[2020-06-15T22:06:30,454][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}
```
Could you help me?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 13, 2020, 9:13pm UTC](https://discuss.elastic.co/t/calculating-new-fields-using-ruby-filter/237163/4 "2020-07-13T21:13:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.