Enable Logstash to Talk with Outside Server?

Hi Logstash,

While processing data records, I need my Logstash instance (ver 7.7.1) to send a copy of the entire data record to an outside server. The server does some processing, then sends back a one-string response. I thought I should be able to do this via a TCP socket. Here’s a toy example:

If my original data record is this:

  Officer_Name   Race       IQ    Rank
  “Spock”        "Vulcan”   180   “Commander”

I need Logstash to send that data record, then wait for the server to respond with:

“USS Enterprise”

And then make the final data record look like this:

Officer_Name   Race       IQ    Rank           Assigned_to
“Spock”        “Vulcan”   180   “Commander”    “USS Enterprise”

Should be simple. I was hoping to do that whole TCP socket thing within Logstash’s filter section. Here’s what I’m tinkering with:

01 filter {
51     ruby {
52       require 'socket'
53       socket = TCPSocket.new("", 12345)
54       socket.write "    "+">>>"+(event.to_hash).to_s+"<<<"
55       response = socket.recv(10000000)
56       event.set("Assigned_to", response)
57     }
58  }

The problem is that LG doesn’t understand line 52; “require ‘socket’”. Here’s the error message in the log:

[2020-08-09T18:47:44,562][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\
t\\r\\n], \"#\", \"=>\" at line 52, column 15 (byte 1023) after filter {… \n\n  ruby {\n    require ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:58:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:66:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:28:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:27:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:181:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:67:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:43:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:342:in `block in converge_state'"]}

So… what can I do? I could do this:

01 filter {
51     ruby {
52       # Bounce all data records through this script:
53       path => "/usr/share/logstash/config/myRubyScript.rb"
54     }
55 }

And put the above code into the “MyRubyScript.rb” file. But it seems like a better solution to try and keep the code in the LG config file. One less node in the chain to possibly fail.

But, having said that, I’m open to possibilities. Ultimately, I need LG to send a data record to an external server, wait, then incorporate the server’s response into the record. Any suggestions are welcome.


 ruby {
    init = "
       require 'socket'
    code => '
        socket = TCPSocket.new("", 12345)
        socket.write "    "+">>>"+(event.to_hash).to_s+"<<<"
        response = socket.recv(10000000)
        event.set("Assigned_to", response)

WOw, thanks Badger, you really know your stuff. Much appreciated.

Minor syntax nitpick for anyone following this post:

The second line should be:


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.