Logstash mutating/renaming with glob

Elastic Stack Logstash
1

I try to efficiently rename pattern in logstash. So if the json field comes as instrument_network_a it will be renamed to network.labels.a, it it comes as instrument_network_b it will be renamed to network.labels.b and so on.
Really what I thought it would be some mutate { gsub { } } expression but can't figure out how to make it to work with glob (*). So I try ruby filter instead.
I tried to wrap gsub inside the event to hash conversion but I have a problems to make it executing.
Below are three cases of different approaches:

Case 1 taken from StackOverflow but it causes logstash to break execution.

  1 - pipeline.id: amq
  2   config.string: |
  3     input { stdin { } }
  4
  5     filter {
  6         json { source => "message" }
  7         ruby { code => ' event.to_hash.each_pair { |k, v|
  8                   if !!(/instrument_network_*/  =~ k)
  9                         newkey = k.gsub(/^instrument_network_/, 'network.labels.')
 10                         event.set(newkey, v);
 11                         event.remove(k)
 12                   end }'
 13              }
 14     }
 15
 16     output { stdout { codec => rubydebug } }

When I run it it just stops:

$ /usr/share/logstash/bin/logstash --path.settings . --path.data data -l . --log.level debug
Sending Logstash's logs to . which is now configured via log4j2.properties
$

I logstash-plain.log the error shows as:

[2019-01-26T20:14:26,015][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:amq, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, } at line 7, column 62 (byte 229) after filter {\n    json { source => \"message\" }\n    ruby { code => ' event.to_hash.each_pair { |k, v|\n              if !!(/instrument_network_*/  =~ k)\n                    newkey = k.gsub(/^instrument_network_/, '", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:49:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:167:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:305:in `block in converge_state'"]}

which I don't really see any clue why it does not work.

I have built another parsing solution base on string

  1 - pipeline.id: amq
  2   config.string: |
  3     input { stdin { } }
  4
  5     filter {
  6         json { source => "message" }
  7         ruby { code => ' event.to_hash.each_pair { |k, v|
  8                   if !!(/instrument_network_*/  =~ k)
  9                         newkey = "#{["network.labels", k.split("_", 3)[2]].join(".")}"
 10                         event.set(newkey, v);
 11                         event.remove(k)
 12                   end }'
 13              }
 14     }
 15
 16     output { stdout { codec => rubydebug } }

with the output:

$ /usr/share/logstash/bin/logstash --path.settings . --path.data data -l . --log.level debug
Sending Logstash's logs to . which is now configured via log4j2.properties
The stdin plugin is now waiting for input:
{"instrument_network_a":false, "instrument_network_b":true, "instrument_network":false}
{
    "network.labels.a" => false,
            "@version" => "1",
          "@timestamp" => 2019-01-26T20:08:52.692Z,
    "network.labels.b" => true,
     "network.labels." => false,
                "host" => "***",
             "message" => "{\"instrument_network_a\":false, \"instrument_network_b\":true, \"instrument_network\":false}"
}

but it feels neither clear to read not efficient to work

  1. What would be the most efficient way to built "mutate" filter using glob?
  2. Is there any way to measure or time execution of the filter?
  3. Why rsub does not work in this case. As far as I read the docs and check syntax, I understand it should work.
  4. Ideally I would like to eliminate network.labels without the name of the instrument. So network.labels.a is accepted but network.labels. really is not. Is there a simple/efficient way to incorporate such behaviour?

All advise well appreciated.

2

OK, so it blows up when it hits the single quote used to quote 'network.labels.'. You are using single quotes for code => '...', so either use double quotes around "network.labels." or escape the single quotes.

Do not use period in field names. It works most of the time, right up to the point where things break.

4

the escape does not seems to work:

[ERROR][logstash.agent           ] Failed to execute action {:id=>:amq, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<amq>, action_result: false", :backtrace=>nil}
[2019-01-26T21:49:50,518][ERROR][org.logstash.Logstash    ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SyntaxError) (ruby filter code):4: syntax error, unexpected null
                    newkey = k.gsub(/^instrument_network_/, \'network.labels.\')
5

Looks like double quote " does the job although I read that really single one should be used in ruby gsub. Maybe I am confused now. Hope it will not cause problems in production.

6

Thanks @Badger. When you write "do not use period in field names ... things break " what do you mean by that? I see you have good knowledge of logstash and and honestly your comment made me worry.
I follow elastic ECS and all of it is built on periods in the names.
What should I be prepared for?

7

In 5.0 or later dots are allowed in field names. However, I know I've seen things break when you use them. For example, as I recall it, sometimes machine learning would replace a field name containing dots with something like "field#17".

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.