Logstash :: What Would a NULL Returned from an External Program Mean?

Elastic Stack Logstash
1

Hi Logstash Gurus,

I am using my Logstash filter{} configuration to use Ruby to “bounce” all my LS records off an external C program via a TCP socket. The C program accepts every record, does some processing, and then returns an extra piece of data – a string – which LS stores as a new field within the record. I’ll explain in more detail below, if what I’ve written so far is confusing.

But my question is this: When the C program replies to LS, it should always send back a string. Always. Yet, when I inspect my LG records after the filter{} section is done with them, I notice that some records have an value of “null” in the new, added field. What does the “null” mean?

Here’s the more detailed description:

I’m running LS 7.7.0, running on an Ubuntu machine. I’m using LS as a network data collector, meaning that my LS gets a lot of raw network data from our routers and switches. Here’s a few data records for example, viewed from ElasticSearch SQL:

   HostA       |   HostB       |tcpPort |totalBytes
---------------+---------------+--------+----------
168.161.114.52 |168.161.114.130|23      |231626752
168.161.114.52 |168.161.114.130|23      |88975360
168.161.114.52 |168.161.114.130|23      |45131776

There are actually a lot more data fields per record, but you get the idea. The whole setup works great. But I need to know what network application is at work here, and LS can’t analyze that for me. This is why I “bounce” every record off that external C program. Here’s my filter{} config:

filter {

  ruby {
    init => "
      require 'socket'
    "

    code => '
      socket = TCPSocket.new("192.168.3.1", 12345)
      socket.write (event.to_hash).to_s
      response = socket.recv(10000000)
      event.set("Application", response)
    '
  }

My C program is listening on host 192.168.3.1, TCP port 12345. When it is up and running, I can see this:

   HostA       |   HostB       |tcpPort|totalBytes|Application
---------------+---------------+-------|----------+----------+
168.161.114.52 |168.161.114.130|23     |231626752 |FTP_DATA   
168.161.114.52 |168.161.114.130|23     |88975360  |FTP_DATA

…but every now and then, I see this:

   HostA       |   HostB       |tcpPort|totalBytes|Application
---------------+---------------+-------+----------+-----------+
168.161.114.52 |168.161.114.130|23     |45131776  |null

Uh-oh. See that "null"? The value of “Application” should never be “null”. In the event my C program can’t analyze the record and recognize the application, it should return a default string, not NULL.

So what could a “null” in the “Application” field indicate here?

I’m guessing that LS sent a data record to the C program, but then did not get a response before a TCP timeout, or something. If the TCP socket was timing out, wouldn’t Ruby throw an exception or something? Could I code to catch that exception?

I also suppose its possible that the C program is returning NULL. But I think this is unlikely.

I realize I am asking an open-ended question, and apologize in advance. Any observations or advice is appreciated. Thank you.

2

EDIT: I've rewritten my post here. Please disregard/delete this one.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.