# My ruby filter doesn't work

**URL:** <https://discuss.elastic.co/t/my-ruby-filter-doesnt-work/258439>\
**Category:** Logstash\
**Created:** [December 11, 2020, 4:55pm UTC](https://discuss.elastic.co/t/my-ruby-filter-doesnt-work/258439 "2020-12-11T16:55:51Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![olegans](https://avatars.discourse-cdn.com/v4/letter/o/5fc32e/32.png) [@olegans](https://discuss.elastic.co/u/olegans)\
**Post date:** [December 11, 2020, 4:55pm UTC](https://discuss.elastic.co/t/my-ruby-filter-doesnt-work/258439/1 "2020-12-11T16:55:52Z")

</div>

I am trying to use a ruby ​​filter to process incoming json that I receive in the logs. At the beginning, using a grok filter, I receive json, then I try to process it in a ruby ​​filter

```auto
    grok {
        match => { "logdata" => ["(spx.responce=%{GREEDYDATA:spx_responce}(\n|;))"]}
    }
    
    ruby {
        # json handler
        init => "require 'json'"
        
        code => "
            i_json = event.get('spx_responce')  
            json = JSON.parse(i_json)
            result = json.map do |key, value|
              if value.is_a?(Array)
                value.map { |v| {v.first.last => v['value']} }
              else
                { key => value }
              end
            end      
            o_json_k = result.flatten.flat_map(&:to_a).to_h.keys
            o_json_v = result.flatten.flat_map(&:to_a).to_h.valies
        "
    }
    
    mutate { 
        add_field => { "tags.key" => "%{o_json_k}" } 
    }
    mutate { 
        add_field => { "tags.value" => "%{o_json_v}" } 
    }

```

When I try to parse the message, I get

```auto
[2020-12-11T19:36:18,073][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}
[2020-12-11T19:36:30,949][ERROR][logstash.filters.ruby][main][c4c7a7bbb68990f1f28a0d79b0147b35abb6af042319e8005e2a71f6dff4ecc7] Ruby exception occurred: undefined method `valies' for #<Hash:0xd487846>
[2020-12-11T19:36:31,046][INFO][logstash.outputs.file][main][8fe97c8a427d4d855743e6897db431043e063dc521e260577618106baf8efe7d] Opening file {:path=>"/home/oleg/log/1.log"}
[2020-12-11T19:36:52,609][INFO][logstash.outputs.file][main][8fe97c8a427d4d855743e6897db431043e063dc521e260577618106baf8efe7d] Closing file /home/oleg/log/1.log

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [December 11, 2020, 5:04pm UTC](https://discuss.elastic.co/t/my-ruby-filter-doesnt-work/258439/2 "2020-12-11T17:04:31Z")

</div>

> [@olegans](#):
>
> `result.flatten.flat_map(&:to_a).to_h.valies`

That should be values, not valies.

In the ruby filter you create local variables called o\_json\_k and o\_json\_v, but they go out of scope and cease to exist when the ruby filter finishes processing an event. If you want to add them to the event then remove the mutate+add\_field filters and add this to the code option of the ruby filter

```
event.set("tags.key", o_json_k)
event.set("tags.value", o_json_v)

```

---

<div class="post-metadata">

**Author:** ![olegans](https://avatars.discourse-cdn.com/v4/letter/o/5fc32e/32.png) [@olegans](https://discuss.elastic.co/u/olegans)\
**Post date:** [December 11, 2020, 5:18pm UTC](https://discuss.elastic.co/t/my-ruby-filter-doesnt-work/258439/3 "2020-12-11T17:18:18Z")

</div>

Thank you so much!

I remomed `mutate` lines and added event.set. It looks like

```auto
 grok {
        match => { "logdata" => ["(spx.responce=%{GREEDYDATA:spx_responce}(\n|;))"]}
    }
    
    ruby {
        # json handler
        init => "require 'json'"
        
        code => "
            i_json = event.get('spx_responce')  
            json = JSON.parse(i_json)
            result = json.map do |key, value|
              if value.is_a?(Array)
                value.map { |v| {v.first.last => v['value']} }
              else
                { key => value }
              end
            end      
            o_json_k = result.flatten.flat_map(&:to_a).to_h.keys
            o_json_v = result.flatten.flat_map(&:to_a).to_h.values
            event.set("tags.key", o_json_k)
            event.set("tags.value", o_json_v)
        "
    }
    
    prune {
        whitelist_names => ["^hostname$","^domain$","^created$","^ms_duration$","^setuptime$","^call_id$","^code$","^reason$","^ip_src$", "^ip_dst$", "^ip_sdp$", "^sip_from$","^sip_to$","^isup_4xx_fs_cause$","^isup_bye_mss_cause$","^sip_bye_fs_reason$","^spx_req$","^tags.key$","^tags.value$"] 
    }

```

But now I get some mistake when I logstah is starting

```auto
[2020-12-11T20:13:40,183][INFO][logstash.runner] Logstash shut down.
[2020-12-11T20:13:40,198][ERROR][org.logstash.Logstash] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2020-12-11T20:14:00,686][INFO][logstash.runner] Starting Logstash {"logstash.version"=>"7.10.0", "jruby.version"=>"jruby 9.2.13.0 (2.5.7) 2020-08-03 9a89c94bcc OpenJDK 64-Bit Server VM 11.0.8+10 on 11.0.8+10 +indy +jit [linux-x86_64]"}
[2020-12-11T20:14:03,469][ERROR][logstash.agent] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [\\t\\r\\n], \"#\", \"{\", \"}\" at line 80, column 24 (byte 2341) after filter{\n grok {\n match => { \"message\" => [\"%{SYSLOGTIMESTAMP:date}%{SPACE}%{HOSTNAME:hostname}%{SPACE}/usr/sbin/opensips\\[%{POSINT}\\]: ACC: call %{WORD:call_status}: %{GREEDYDATA:logdata}\"]}\n }\n if [call_status] !~ /.+/ {\n drop {}\n }\n grok {\n match=>{ \"logdata\" => [\"(created=%{NONNEGINT:created})\"]}\n }\n grok {\n match=>{ \"logdata\" => [\"(ms_duration=%{NONNEGINT:ms_duration})\"]}\n }\n grok {\n match=>{ \"logdata\" => [\"(setuptime=%{NONNEGINT:setuptime})\"]}\n }\n grok {\n match=>{ \"logdata\" => [\"(call_id=%{GREEDYDATA:call_id};code)\"]}\n }\n grok {\n match=>{ \"logdata\" => [\"(code=%{NONNEGINT:code})\"]}\n }\n grok {\n match=>{ \"logdata\" => [\"(reason=%{DATA:reason};ip)\"]}\n }\n grok {\n match => { \"logdata\" => [\"(ip.src=%{IP:ip_src})\"]}\n }\n grok {\n match => { \"logdata\" => [\"(ip.dst=%{IP:ip_dst})\"]}\n }\n grok {\n match => { \"logdata\" => [\"(ip.sdp=%{IP:ip_sdp})\"]}\n }\n grok {\n match => { \"logdata\" => [\"(sip.from=(\\+|.*)%{NONNEGINT:sip_from};)\"]}\n }\n grok {\n match => { \"logdata\" => [\"(sip.to=%{NONNEGINT:sip_to};)\"]}\n }\n grok {\n match => { \"logdata\" => [\"(isup.4xx_fs.cause=%{NONNEGINT:isup_4xx_fs_cause})\"]}\n }\n grok {\n match => { \"logdata\" => [\"(isup.bye_mss.cause=%{NONNEGINT:isup_bye_mss_cause};)\"]}\n }\n grok {\n match => { \"logdata\" => [\"(sip.bye_fs.reason=%{DATA:sip_bye_fs_reason};spx)\"]}\n }\n grok {\n match => { \"logdata\" => [\"(spx.req=\\{%{DATA:spx_req}\\})\"]}\n }\n grok {\n match => { \"logdata\" => [\"(spx.responce=%{GREEDYDATA:spx_responce}(\\n|;))\"]}\n }\n \n ruby {\n # json handler\n init => \"require 'json'\"\n \n code => \"\n i_json = event.get('spx_responce') \n json = JSON.parse(i_json)\n result = json.map do |key, value|\n if value.is_a?(Array)\n value.map { |v| {v.first.last => v['value']} }\n else\n { key => value }\n end\n end \n o_json_k = result.flatten.flat_map(&:to_a).to_h.keys\n o_json_v = result.flatten.flat_map(&:to_a).to_h.values\n event.set(\"", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:184:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:365:in `block in converge_state'"]}
[2020-12-11T20:14:03,819][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}
[2020-12-11T20:14:08,746][INFO][logstash.runner] Logstash shut down.

```

---

<div class="post-metadata">

**Author:** ![olegans](https://avatars.discourse-cdn.com/v4/letter/o/5fc32e/32.png) [@olegans](https://discuss.elastic.co/u/olegans)\
**Post date:** [December 11, 2020, 5:40pm UTC](https://discuss.elastic.co/t/my-ruby-filter-doesnt-work/258439/4 "2020-12-11T17:40:38Z")

</div>

I did some changes.

was

```auto
event.set ("tags.key", o_json_k)
event.set ("tags.value", o_json_v) 

```

now

```auto
event.set ('tags.key', o_json_k)
event.set ('tags.value', o_json_v)

```

---

<div class="post-metadata">

**Author:** ![olegans](https://avatars.discourse-cdn.com/v4/letter/o/5fc32e/32.png) [@olegans](https://discuss.elastic.co/u/olegans)\
**Post date:** [December 11, 2020, 5:42pm UTC](https://discuss.elastic.co/t/my-ruby-filter-doesnt-work/258439/5 "2020-12-11T17:42:27Z")

</div>

And now everything works

```auto
    grok {
        match => { "logdata" => ["(spx.responce=%{GREEDYDATA:spx_responce}(\n|;))"]}
    }
    
    ruby {
        # json handler
        init => "require 'json'"
        
        code => "
            i_json = event.get('spx_responce')  
            json = JSON.parse(i_json)
            result = json.map do |key, value|
              if value.is_a?(Array)
                value.map { |v| {v.first.last => v['value']} }
              else
                { key => value }
              end
            end      
            o_json_k = result.flatten.flat_map(&:to_a).to_h.keys
            o_json_v = result.flatten.flat_map(&:to_a).to_h.values
            event.set('tags.key', o_json_k)
            event.set('tags.value', o_json_v)
        "
    }
    
    prune {
        whitelist_names => ["^hostname$","^domain$","^created$","^ms_duration$","^setuptime$","^call_id$","^code$","^reason$","^ip_src$", "^ip_dst$", "^ip_sdp$", "^sip_from$","^sip_to$","^isup_4xx_fs_cause$","^isup_bye_mss_cause$","^sip_bye_fs_reason$","^spx_req$","^tags.key$","^tags.value$"] 
    }

```

What is the difference between these quotes?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [December 11, 2020, 6:22pm UTC](https://discuss.elastic.co/t/my-ruby-filter-doesnt-work/258439/6 "2020-12-11T18:22:39Z")

</div>

Personally I always use

```
code => ' ... '

```

rather than

```
code => " ... "

```

so that I can use string interpolation if I need it. But it is your choice. If you use double quotes around the code then use single quotes within it, if you use single quotes around then code then use double quotes within it.

In logstash and ruby (with the exception of string interpolation) single and double quotes tend to be interchangeable.

---

<div class="post-metadata">

**Author:** ![olegans](https://avatars.discourse-cdn.com/v4/letter/o/5fc32e/32.png) [@olegans](https://discuss.elastic.co/u/olegans)\
**Post date:** [December 11, 2020, 7:01pm UTC](https://discuss.elastic.co/t/my-ruby-filter-doesnt-work/258439/7 "2020-12-11T19:01:01Z")

</div>

Thank you for your help.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 8, 2021, 7:01pm UTC](https://discuss.elastic.co/t/my-ruby-filter-doesnt-work/258439/8 "2021-01-08T19:01:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.