My ruby filter doesn't work

olegans · December 11, 2020, 4:55pm

I am trying to use a ruby filter to process incoming json that I receive in the logs. At the beginning, using a grok filter, I receive json, then I try to process it in a ruby filter

    grok {
        match => { "logdata" => ["(spx.responce=%{GREEDYDATA:spx_responce}(\n|;))"]}
    }
    
    ruby {
        # json handler
        init => "require 'json'"
        
        code => "
            i_json = event.get('spx_responce')  
            json = JSON.parse(i_json)
            result = json.map do |key, value|
              if value.is_a?(Array)
                value.map { |v| {v.first.last => v['value']} }
              else
                { key => value }
              end
            end      
            o_json_k = result.flatten.flat_map(&:to_a).to_h.keys
            o_json_v = result.flatten.flat_map(&:to_a).to_h.valies
        "
    }
    
    mutate { 
        add_field => { "tags.key" => "%{o_json_k}" } 
    }
    mutate { 
        add_field => { "tags.value" => "%{o_json_v}" } 
    }

When I try to parse the message, I get

[2020-12-11T19:36:18,073][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2020-12-11T19:36:30,949][ERROR][logstash.filters.ruby    ][main][c4c7a7bbb68990f1f28a0d79b0147b35abb6af042319e8005e2a71f6dff4ecc7] Ruby exception occurred: undefined method `valies' for #<Hash:0xd487846>
[2020-12-11T19:36:31,046][INFO ][logstash.outputs.file    ][main][8fe97c8a427d4d855743e6897db431043e063dc521e260577618106baf8efe7d] Opening file {:path=>"/home/oleg/log/1.log"}
[2020-12-11T19:36:52,609][INFO ][logstash.outputs.file    ][main][8fe97c8a427d4d855743e6897db431043e063dc521e260577618106baf8efe7d] Closing file /home/oleg/log/1.log

Badger · December 11, 2020, 5:04pm

That should be values, not valies.

In the ruby filter you create local variables called o_json_k and o_json_v, but they go out of scope and cease to exist when the ruby filter finishes processing an event. If you want to add them to the event then remove the mutate+add_field filters and add this to the code option of the ruby filter

event.set("tags.key", o_json_k)
event.set("tags.value", o_json_v)

olegans · December 11, 2020, 5:18pm

Thank you so much!

I remomed mutate lines and added event.set. It looks like

 grok {
        match => { "logdata" => ["(spx.responce=%{GREEDYDATA:spx_responce}(\n|;))"]}
    }
    
    ruby {
        # json handler
        init => "require 'json'"
        
        code => "
            i_json = event.get('spx_responce')  
            json = JSON.parse(i_json)
            result = json.map do |key, value|
              if value.is_a?(Array)
                value.map { |v| {v.first.last => v['value']} }
              else
                { key => value }
              end
            end      
            o_json_k = result.flatten.flat_map(&:to_a).to_h.keys
            o_json_v = result.flatten.flat_map(&:to_a).to_h.values
            event.set("tags.key", o_json_k)
            event.set("tags.value", o_json_v)
        "
    }
    
    prune {
        whitelist_names => [ "^hostname$","^domain$","^created$","^ms_duration$","^setuptime$","^call_id$","^code$","^reason$","^ip_src$", "^ip_dst$", "^ip_sdp$", "^sip_from$","^sip_to$","^isup_4xx_fs_cause$","^isup_bye_mss_cause$","^sip_bye_fs_reason$","^spx_req$","^tags.key$","^tags.value$" ] 
    }

But now I get some mistake when I logstah is starting

[2020-12-11T20:13:40,183][INFO ][logstash.runner          ] Logstash shut down.
[2020-12-11T20:13:40,198][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2020-12-11T20:14:00,686][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.10.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +indy +jit [linux-x86_64]"}
[2020-12-11T20:14:03,469][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"{\", \"}\" at line 80, column 24 (byte 2341) after filter{\n    grok {\n        match => { \"message\" => [\"%{SYSLOGTIMESTAMP:date}%{SPACE}%{HOSTNAME:hostname}%{SPACE}/usr/sbin/opensips\\[%{POSINT}\\]: ACC: call %{WORD:call_status}: %{GREEDYDATA:logdata}\"]}\n    }\n    if [call_status] !~ /.+/ {\n        drop {}\n    }\n    grok {\n        match=>{ \"logdata\" => [\"(created=%{NONNEGINT:created})\"]}\n    }\n    grok {\n        match=>{ \"logdata\" => [\"(ms_duration=%{NONNEGINT:ms_duration})\"]}\n    }\n    grok {\n        match=>{ \"logdata\" => [\"(setuptime=%{NONNEGINT:setuptime})\"]}\n    }\n    grok {\n        match=>{ \"logdata\" => [\"(call_id=%{GREEDYDATA:call_id};code)\"]}\n    }\n    grok {\n        match=>{ \"logdata\" => [\"(code=%{NONNEGINT:code})\"]}\n    }\n    grok {\n        match=>{ \"logdata\" => [\"(reason=%{DATA:reason};ip)\"]}\n    }\n    grok {\n        match => { \"logdata\" => [\"(ip.src=%{IP:ip_src})\"]}\n    }\n    grok {\n        match => { \"logdata\" => [\"(ip.dst=%{IP:ip_dst})\"]}\n    }\n    grok {\n        match => { \"logdata\" => [\"(ip.sdp=%{IP:ip_sdp})\"]}\n    }\n    grok {\n        match => { \"logdata\" => [\"(sip.from=(\\+|.*)%{NONNEGINT:sip_from};)\"]}\n    }\n    grok {\n        match => { \"logdata\" => [\"(sip.to=%{NONNEGINT:sip_to};)\"]}\n    }\n    grok {\n        match => { \"logdata\" => [\"(isup.4xx_fs.cause=%{NONNEGINT:isup_4xx_fs_cause})\"]}\n    }\n    grok {\n        match => { \"logdata\" => [\"(isup.bye_mss.cause=%{NONNEGINT:isup_bye_mss_cause};)\"]}\n    }\n    grok {\n        match => { \"logdata\" => [\"(sip.bye_fs.reason=%{DATA:sip_bye_fs_reason};spx)\"]}\n    }\n    grok {\n        match => { \"logdata\" => [\"(spx.req=\\{%{DATA:spx_req}\\})\"]}\n    }\n    grok {\n        match => { \"logdata\" => [\"(spx.responce=%{GREEDYDATA:spx_responce}(\\n|;))\"]}\n    }\n    \n    ruby {\n        # json handler\n        init => \"require 'json'\"\n        \n        code => \"\n            i_json = event.get('spx_responce')  \n            json = JSON.parse(i_json)\n            result = json.map do |key, value|\n              if value.is_a?(Array)\n                value.map { |v| {v.first.last => v['value']} }\n              else\n                { key => value }\n              end\n            end      \n            o_json_k = result.flatten.flat_map(&:to_a).to_h.keys\n            o_json_v = result.flatten.flat_map(&:to_a).to_h.values\n            event.set(\"", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:365:in `block in converge_state'"]}
[2020-12-11T20:14:03,819][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2020-12-11T20:14:08,746][INFO ][logstash.runner          ] Logstash shut down.

olegans · December 11, 2020, 5:40pm

I did some changes.

was

event.set ("tags.key", o_json_k)
event.set ("tags.value", o_json_v)

now

event.set ('tags.key', o_json_k)
event.set ('tags.value', o_json_v)

olegans · December 11, 2020, 5:42pm

And now everything works

    grok {
        match => { "logdata" => ["(spx.responce=%{GREEDYDATA:spx_responce}(\n|;))"]}
    }
    
    ruby {
        # json handler
        init => "require 'json'"
        
        code => "
            i_json = event.get('spx_responce')  
            json = JSON.parse(i_json)
            result = json.map do |key, value|
              if value.is_a?(Array)
                value.map { |v| {v.first.last => v['value']} }
              else
                { key => value }
              end
            end      
            o_json_k = result.flatten.flat_map(&:to_a).to_h.keys
            o_json_v = result.flatten.flat_map(&:to_a).to_h.values
            event.set('tags.key', o_json_k)
            event.set('tags.value', o_json_v)
        "
    }
    
    prune {
        whitelist_names => [ "^hostname$","^domain$","^created$","^ms_duration$","^setuptime$","^call_id$","^code$","^reason$","^ip_src$", "^ip_dst$", "^ip_sdp$", "^sip_from$","^sip_to$","^isup_4xx_fs_cause$","^isup_bye_mss_cause$","^sip_bye_fs_reason$","^spx_req$","^tags.key$","^tags.value$" ] 
    }

What is the difference between these quotes?

Badger · December 11, 2020, 6:22pm

Personally I always use

code => ' ... '

rather than

code => " ... "

so that I can use string interpolation if I need it. But it is your choice. If you use double quotes around the code then use single quotes within it, if you use single quotes around then code then use double quotes within it.

In logstash and ruby (with the exception of string interpolation) single and double quotes tend to be interchangeable.

olegans · December 11, 2020, 7:01pm

Thank you for your help.

system · January 8, 2021, 7:01pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.