Hi Logstash,
While processing data records, I need my Logstash instance (ver 7.7.1) to send a copy of the entire data record to an outside server. The server does some processing, then sends back a one-string response. I thought I should be able to do this via a TCP socket. Here’s a toy example:
If my original data record is this:
Officer_Name Race IQ Rank
--------------------------------------------
“Spock” "Vulcan” 180 “Commander”
I need Logstash to send that data record, then wait for the server to respond with:
“USS Enterprise”
And then make the final data record look like this:
Officer_Name Race IQ Rank Assigned_to
---------------------------------------------------------------
“Spock” “Vulcan” 180 “Commander” “USS Enterprise”
Should be simple. I was hoping to do that whole TCP socket thing within Logstash’s filter section. Here’s what I’m tinkering with:
01 filter {
...more...
51 ruby {
52 require 'socket'
53 socket = TCPSocket.new("192.168.1.100", 12345)
54 socket.write " "+">>>"+(event.to_hash).to_s+"<<<"
55 response = socket.recv(10000000)
56 event.set("Assigned_to", response)
57 }
58 }
The problem is that LG doesn’t understand line 52; “require ‘socket’”. Here’s the error message in the log:
[2020-08-09T18:47:44,562][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\
t\\r\\n], \"#\", \"=>\" at line 52, column 15 (byte 1023) after filter {… \n\n ruby {\n require ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:58:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:66:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:28:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:27:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:181:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:67:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:43:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:342:in `block in converge_state'"]}
So… what can I do? I could do this:
01 filter {
...more...
51 ruby {
52 # Bounce all data records through this script:
53 path => "/usr/share/logstash/config/myRubyScript.rb"
54 }
55 }
And put the above code into the “MyRubyScript.rb
” file. But it seems like a better solution to try and keep the code in the LG config file. One less node in the chain to possibly fail.
But, having said that, I’m open to possibilities. Ultimately, I need LG to send a data record to an external server, wait, then incorporate the server’s response into the record. Any suggestions are welcome.