My ruby filter doesn't work

I am trying to use a ruby ​​filter to process incoming json that I receive in the logs. At the beginning, using a grok filter, I receive json, then I try to process it in a ruby ​​filter

    grok {
        match => { "logdata" => ["(spx.responce=%{GREEDYDATA:spx_responce}(\n|;))"]}
    }
    
    ruby {
        # json handler
        init => "require 'json'"
        
        code => "
            i_json = event.get('spx_responce')  
            json = JSON.parse(i_json)
            result = json.map do |key, value|
              if value.is_a?(Array)
                value.map { |v| {v.first.last => v['value']} }
              else
                { key => value }
              end
            end      
            o_json_k = result.flatten.flat_map(&:to_a).to_h.keys
            o_json_v = result.flatten.flat_map(&:to_a).to_h.valies
        "
    }
    
    mutate { 
        add_field => { "tags.key" => "%{o_json_k}" } 
    }
    mutate { 
        add_field => { "tags.value" => "%{o_json_v}" } 
    }

When I try to parse the message, I get

[2020-12-11T19:36:18,073][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2020-12-11T19:36:30,949][ERROR][logstash.filters.ruby    ][main][c4c7a7bbb68990f1f28a0d79b0147b35abb6af042319e8005e2a71f6dff4ecc7] Ruby exception occurred: undefined method `valies' for #<Hash:0xd487846>
[2020-12-11T19:36:31,046][INFO ][logstash.outputs.file    ][main][8fe97c8a427d4d855743e6897db431043e063dc521e260577618106baf8efe7d] Opening file {:path=>"/home/oleg/log/1.log"}
[2020-12-11T19:36:52,609][INFO ][logstash.outputs.file    ][main][8fe97c8a427d4d855743e6897db431043e063dc521e260577618106baf8efe7d] Closing file /home/oleg/log/1.log

That should be values, not valies.

In the ruby filter you create local variables called o_json_k and o_json_v, but they go out of scope and cease to exist when the ruby filter finishes processing an event. If you want to add them to the event then remove the mutate+add_field filters and add this to the code option of the ruby filter

event.set("tags.key", o_json_k)
event.set("tags.value", o_json_v)

Thank you so much!

I remomed mutate lines and added event.set. It looks like

 grok {
        match => { "logdata" => ["(spx.responce=%{GREEDYDATA:spx_responce}(\n|;))"]}
    }
    
    ruby {
        # json handler
        init => "require 'json'"
        
        code => "
            i_json = event.get('spx_responce')  
            json = JSON.parse(i_json)
            result = json.map do |key, value|
              if value.is_a?(Array)
                value.map { |v| {v.first.last => v['value']} }
              else
                { key => value }
              end
            end      
            o_json_k = result.flatten.flat_map(&:to_a).to_h.keys
            o_json_v = result.flatten.flat_map(&:to_a).to_h.values
            event.set("tags.key", o_json_k)
            event.set("tags.value", o_json_v)
        "
    }
    
    prune {
        whitelist_names => [ "^hostname$","^domain$","^created$","^ms_duration$","^setuptime$","^call_id$","^code$","^reason$","^ip_src$", "^ip_dst$", "^ip_sdp$", "^sip_from$","^sip_to$","^isup_4xx_fs_cause$","^isup_bye_mss_cause$","^sip_bye_fs_reason$","^spx_req$","^tags.key$","^tags.value$" ] 
    }

But now I get some mistake when I logstah is starting

[2020-12-11T20:13:40,183][INFO ][logstash.runner          ] Logstash shut down.
[2020-12-11T20:13:40,198][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2020-12-11T20:14:00,686][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.10.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +indy +jit [linux-x86_64]"}
[2020-12-11T20:14:03,469][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"{\", \"}\" at line 80, column 24 (byte 2341) after filter{\n    grok {\n        match => { \"message\" => [\"%{SYSLOGTIMESTAMP:date}%{SPACE}%{HOSTNAME:hostname}%{SPACE}/usr/sbin/opensips\\[%{POSINT}\\]: ACC: call %{WORD:call_status}: %{GREEDYDATA:logdata}\"]}\n    }\n    if [call_status] !~ /.+/ {\n        drop {}\n    }\n    grok {\n        match=>{ \"logdata\" => [\"(created=%{NONNEGINT:created})\"]}\n    }\n    grok {\n        match=>{ \"logdata\" => [\"(ms_duration=%{NONNEGINT:ms_duration})\"]}\n    }\n    grok {\n        match=>{ \"logdata\" => [\"(setuptime=%{NONNEGINT:setuptime})\"]}\n    }\n    grok {\n        match=>{ \"logdata\" => [\"(call_id=%{GREEDYDATA:call_id};code)\"]}\n    }\n    grok {\n        match=>{ \"logdata\" => [\"(code=%{NONNEGINT:code})\"]}\n    }\n    grok {\n        match=>{ \"logdata\" => [\"(reason=%{DATA:reason};ip)\"]}\n    }\n    grok {\n        match => { \"logdata\" => [\"(ip.src=%{IP:ip_src})\"]}\n    }\n    grok {\n        match => { \"logdata\" => [\"(ip.dst=%{IP:ip_dst})\"]}\n    }\n    grok {\n        match => { \"logdata\" => [\"(ip.sdp=%{IP:ip_sdp})\"]}\n    }\n    grok {\n        match => { \"logdata\" => [\"(sip.from=(\\+|.*)%{NONNEGINT:sip_from};)\"]}\n    }\n    grok {\n        match => { \"logdata\" => [\"(sip.to=%{NONNEGINT:sip_to};)\"]}\n    }\n    grok {\n        match => { \"logdata\" => [\"(isup.4xx_fs.cause=%{NONNEGINT:isup_4xx_fs_cause})\"]}\n    }\n    grok {\n        match => { \"logdata\" => [\"(isup.bye_mss.cause=%{NONNEGINT:isup_bye_mss_cause};)\"]}\n    }\n    grok {\n        match => { \"logdata\" => [\"(sip.bye_fs.reason=%{DATA:sip_bye_fs_reason};spx)\"]}\n    }\n    grok {\n        match => { \"logdata\" => [\"(spx.req=\\{%{DATA:spx_req}\\})\"]}\n    }\n    grok {\n        match => { \"logdata\" => [\"(spx.responce=%{GREEDYDATA:spx_responce}(\\n|;))\"]}\n    }\n    \n    ruby {\n        # json handler\n        init => \"require 'json'\"\n        \n        code => \"\n            i_json = event.get('spx_responce')  \n            json = JSON.parse(i_json)\n            result = json.map do |key, value|\n              if value.is_a?(Array)\n                value.map { |v| {v.first.last => v['value']} }\n              else\n                { key => value }\n              end\n            end      \n            o_json_k = result.flatten.flat_map(&:to_a).to_h.keys\n            o_json_v = result.flatten.flat_map(&:to_a).to_h.values\n            event.set(\"", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:365:in `block in converge_state'"]}
[2020-12-11T20:14:03,819][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2020-12-11T20:14:08,746][INFO ][logstash.runner          ] Logstash shut down.

I did some changes.

was

event.set ("tags.key", o_json_k)
event.set ("tags.value", o_json_v) 

now

event.set ('tags.key', o_json_k)
event.set ('tags.value', o_json_v)

And now everything works

    grok {
        match => { "logdata" => ["(spx.responce=%{GREEDYDATA:spx_responce}(\n|;))"]}
    }
    
    ruby {
        # json handler
        init => "require 'json'"
        
        code => "
            i_json = event.get('spx_responce')  
            json = JSON.parse(i_json)
            result = json.map do |key, value|
              if value.is_a?(Array)
                value.map { |v| {v.first.last => v['value']} }
              else
                { key => value }
              end
            end      
            o_json_k = result.flatten.flat_map(&:to_a).to_h.keys
            o_json_v = result.flatten.flat_map(&:to_a).to_h.values
            event.set('tags.key', o_json_k)
            event.set('tags.value', o_json_v)
        "
    }
    
    prune {
        whitelist_names => [ "^hostname$","^domain$","^created$","^ms_duration$","^setuptime$","^call_id$","^code$","^reason$","^ip_src$", "^ip_dst$", "^ip_sdp$", "^sip_from$","^sip_to$","^isup_4xx_fs_cause$","^isup_bye_mss_cause$","^sip_bye_fs_reason$","^spx_req$","^tags.key$","^tags.value$" ] 
    }

What is the difference between these quotes?

Personally I always use

code => ' ... '

rather than

code => " ... "

so that I can use string interpolation if I need it. But it is your choice. If you use double quotes around the code then use single quotes within it, if you use single quotes around then code then use double quotes within it.

In logstash and ruby (with the exception of string interpolation) single and double quotes tend to be interchangeable.

Thank you for your help.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.