Ruby filter to check fields dynamically

[Anusha_Kusanghi]

Hie All,

From the following logs I want to check on condition HasErrorState = true and drop the entire event.
Data field will be single entry and in that we have multiple entries for Scales.

 "Data": [
        {
            "Scales": [
                {
                    "DataId": 1,
                    "CumulsLastCalculated": "/Date(-62135557200000+0100)/",
                    "FiscalMemoryLastUpdated": "/Date(-62135557200000+0100)/",
                    "HasErrorState": true,
                    "IsActive": true,
                    "ScaleName": "KIBANA",                    
                    "TaskInfos": [
                        {
                            "TaskComment": "",
                            "TaskSuffix": ""
                        }
                    ]
                }
            ],
			{
                    "DataId": 2,
                    "CumulsLastCalculated": "/Date(-62135557200000+0100)/",
                    "FiscalMemoryLastUpdated": "/Date(-62135557200000+0100)/",
                    "HasErrorState": true,
                    "IsActive": true,
                    "ScaleName": "KIBANA",                    
                    "TaskInfos": [
                        {
                            "TaskComment": "",
                            "TaskSuffix": ""
                        }
                    ]
                }
        }
    ]

Tried this but its not working as expected:

ruby {
code => '
event.get([Data][0][Scales]).each do |item|
if item["HasErrorState"] == "true"
event.set("ScaleName", "done")
end
end
'
}

[Badger]

@Anusha_Kusanghi

if item["HasErrorState"] == "true"

You are comparing it to a string. I suggest you try

if item["HasErrorState"] == true
    event.cancel
end

[Anusha_Kusanghi]

Hie @Badger ,

Im not sure but if I put that correctly , But in scales it is a collection of arrays , like 0 , 1 ,2 ... entries.
So if the first entry has HasErrorState=true than I have to drop that document/entry to be stored in elastic.

Tried the above solution but seeing a rubyexception.

Is there is any other way to do this ?

[Badger]

Are you saying you want to delete any entries from [Data][0][Scales] which have HasErrorState true but retain the event? What to do if [Data][0][Scales] ends up as an empty array?

[Anusha_Kusanghi]

Yes @Badger ,you are right on deleting and retain part:
Are you saying you want to delete any entries from [Data][0][Scales] which have HasErrorState true but retain the event - YES

If the array is empty we just have to ignore the if condition part as we can't do much in that part.

[Badger]

Please check the JSON that you showed. [Data][0][Scales] is an array that contains a single hash, when the array ends another hash starts. That's not valid JSON.

I suspect you will want something like

    ruby {
        code => '
            scales = event.get("[Data][0][Scales]")
            scales = scales.delete_if { |item| item["HasErrorState"] }
            event.set("[Data][0][Scales]", scales)
        '
    }

[Anusha_Kusanghi]

IT still doesn't work , May be ill try to correct my mistake in first point.

The input here is:

"Data": [
        {
            "Scales": [
                {
                    "DataId": 1,
                    "CumulsLastCalculated": "/Date(-62135557200000+0100)/",
                    "FiscalMemoryLastUpdated": "/Date(-62135557200000+0100)/",
                    "HasErrorState": true,
                    "IsActive": true,
                    "ScaleName": "KIBANA",                    
                    "TaskInfos": [
                        {
                            "TaskComment": "",
                            "TaskSuffix": ""
                        }
                    ]
                },
			{
                    "DataId": 2,
                    "CumulsLastCalculated": "/Date(-62135557200000+0100)/",
                    "FiscalMemoryLastUpdated": "/Date(-62135557200000+0100)/",
                    "HasErrorState": true,
                    "IsActive": true,
                    "ScaleName": "KIBANA",                    
                    "TaskInfos": [
                        {
                            "TaskComment": "",
                            "TaskSuffix": ""
                        }
                    ]
                }
        ]
		}
    ]

So in array Scales have 2 entries where 1st entry has HasErrorState=true , Which I want to store in my Elasticsearch.

And the other entry HasErrorState=flase , Is what I want to drop.

@Badger , Thanks lot for helping out. Ruby is still a new thing for me so

[Badger]

If the data has that format then the ruby filter I posted should work.

[Anusha_Kusanghi]

Error I see is:

[2022-09-06T20:15:11,312][ERROR][logstash.filters.ruby ][main] Ruby exception occurred: undefined method delete_if' for nil:NilClass {:class=>"NoMethodError", :backtrace=>["(ruby filter code):4:in block in filter_method'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:96:in inline_script'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ruby-3.1.8/lib/logstash/filters/ruby.rb:89:in filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159:in do_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:178:in block in multi_filter'", "org/jruby/RubyArray.java:1821:in each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:175:in multi_filter'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:134:in multi_filter'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:300:in block in start_workers'"]}
[2022-09-06T20:20:40,193][INFO ][logstash.javapipeline ][main] Pipeline terminated {"pipeline.id"=>"main"}

[Anusha_Kusanghi]

@Badger

This small correction in deed worked for me:

ruby {
code => '
scales = event.get("[Data][0][Scales]")
scales = scales.delete_if { |item| item["HasErrorState"] == false }
event.set("[Data][0][Scales]", scales)
'
}