Execute program and assign a field its output through ruby filter

[deostroll (arun jayapal)]

My logstash receives JSON messages from a kafka topic. During development phase it is useful if I can read the json in a flattened way. There exists a program called catj which does this. It is an npm binary. You need to install it globally for it to work in the command line.

The usual way to run it is: echo '<some_json_string>' | catj

Note: I have to insert JSON between single-quotes (in the shell).

However, I get error when I set the filter like so:

filter {

  ruby {

    code => 'event.set("field", `echo 'event.get("message")' | catj`)'

  }

}

What is the correct way of writing code here?

[Rahul_Kumar4 (Rahul Kumar)]

1. What error are you getting?

2. Can you give a sample incoming message and what would you expect it to look like after it has been processed please?

[deostroll (arun jayapal)]

It goes something like this:

[2020-03-28T15:53:27,122][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"{\", \"}\" at line 13, column 38 (byte 219) after filter {\n\truby {\n\t\tcode => 'event.set(\"field\", `echo '", :backtrace=>["D:/apps/elk/logstash-7.6.1/logstash-7.6.1/logstash-core/lib/logstash/compiler.rb:47:in `compile_imperative'", "D:/apps/elk/logstash-7.6.1/logstash-7.6.1/logstash-core/lib/logstash/compiler.rb:55:in `compile_graph'", "D:/apps/elk/logstash-7.6.1/logstash-7.6.1/logstash-core/lib/logstash/compiler.rb:17:in `block in compile_sources'", "org/jruby/RubyArray.java:2580:in `map'", "D:/apps/elk/logstash-7.6.1/logstash-7.6.1/logstash-core/lib/logstash/compiler.rb:14:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:161:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "D:/apps/elk/logstash-7.6.1/logstash-7.6.1/logstash-core/lib/logstash/java_pipeline.rb:27:in `initialize'", "D:/apps/elk/logstash-7.6.1/logstash-7.6.1/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "D:/apps/elk/logstash-7.6.1/logstash-7.6.1/logstash-core/lib/logstash/agent.rb:326:in `block in converge_state'"]}

[Badger]

That's not how you fork a command in ruby. Read this.

[deostroll (arun jayapal)]

Found it.

I realized we don't need the single-quotes.

filter {
	ruby {
		code => 'event.set("catj", `echo #{event.get("message")} | catj --no-color`)'
	}
}