Logstash TCP output to logstash TCP input with SSL error

thangld · May 8, 2019, 9:54am

Hi all.
I'm trying to test Logstash TCP input plugin with SSL configuration so I run two logstash instance, one for TCP output and one for TCP input.
These are my settings:
input.conf :

input {
        tcp {
                port => 9443
                mode => "server"
                ssl_enable => true
                ssl_cert => "/home/sf/thangld13/244cert/server.crt"
                ssl_extra_chain_certs => "/home/sf/thangld13/244cert/ca.crt"
                ssl_key => "/home/sf/thangld13/244cert/server.key"
                ssl_verify => true
        }
}
filter {
    if (1 == 2) {
    }
}
output {
    stdout {
        codec => rubydebug
    }
}

output.conf

input {
  file {
    path => "/var/log/auth.log"
    start_position => "beginning"
  }
}
filter {
    if (1 == 2) {
    }
}
output {
  tcp {
    port => 9443
    host => "localhost"
    ssl_enable => true
    ssl_cacert => "/home/sf/thangld13/244cert/ca.crt"
    ssl_key => "/home/sf/thangld13/244cert/client.key"
    ssl_cert => "/home/sf/thangld13/244cert/client.crt"
    ssl_verify => false
  }
}

Then I run

/usr/share/logstash/bin/logstash -f input.conf
/usr/share/logstash/bin/logstash -f output.conf --path.data /home/sf/thangld13/logstash/data/

And the first instance gives me this error:

java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid21649.hprof ...
Heap dump file created [1075541011 bytes in 6.984 secs]
The signal INT is in use by the JVM and will not work correctly on this platform
[ERROR] 2019-05-08 16:46:39.263 [pool-2-thread-10] deadletterqueue - PeriodicPoller: exception {:poller=>#<LogStash::Instrument::PeriodicPoller::DeadLetterQueue:0x207797a6 @agent=#<LogStash::Agent:0x26f4fd33 @dispatcher=#<LogStash::EventDispatcher:0x1373afcc @emitter=#<LogStash::Agent:0x26f4fd33 ...>, @listeners=<Java::JavaUtilConcurrent::CopyOnWriteArraySet:0 []>>, @id_path="/usr/share/logstash/data/uuid", @metric=#...:polling_timeout=>120, :polling_interval=>5, :executed_at=>2019-05-08 16:46:39 +0700}
Error: Your application used more memory than the safety cap of 1G.
Specify -J-Xmx####m to increase it (#### = cap size in MB).
Specify -w for full OutOfMemoryError stack trace

Then It shuts down.
Can anyone tell me what is wrong here?
Thanks!

thangld · May 8, 2019, 10:04am

Update 1: After I set ssl_verify to false for both input and output it gives me this error:

java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid23084.hprof ...
Heap dump file created [1075397956 bytes in 9.455 secs]
Error: Your application used more memory than the safety cap of 1G.
Specify -J-Xmx####m to increase it (#### = cap size in MB).
Specify -w for full OutOfMemoryError stack trace

Update 2: After I set ssl_verify to true for both input and output, it gives me the same error as Update 1.

Badger · May 8, 2019, 12:54pm

The first few lines of /etc/logstash/jvm.options control the initial and maximum heap size. At a minimum you need to increase the maximum heap size.

thangld · May 9, 2019, 7:12am

It still gives out the same error after I increased the maximum heap size.
When I try to add ssl_certificate_authorities to the input config, it gives me this error: [LogStash::Runner] tcp - Unknown setting 'ssl_certificate_authorities' for tcp.
My logstash version is 5.6.16. How can I use CA for this version then?
Thanks.

A_B · May 9, 2019, 7:31am

I have not connected Logstash to Logstash over TCP with SSL but maybe it is similar to how Filebeat works...

I use this for Filebeat (would be similart to your output.conf). I use self-signed SSL certs

ssl:
  certificate_authorities:
  - "/etc/filebeat/ssl/star.example.com"

IIRC that is the intermediate cert ^^

The beats input is configured with this (would be similar to your input.conf)

ssl => true
ssl_certificate => "/etc/logstash/ssl/example.com.crt"
ssl_key => "/etc/logstash/ssl/example.com.key"

Hope that helps.

thangld · May 9, 2019, 7:56am

I think It has successfully verified. It doesn't give any handshake errors. But it doesn't write each event log to the terminal. Instead, when I stop the sending logstash, the receiving logstash gives out all the events at once and as it is just one event...
Sending logstash:

input {
  file {
    path => "/var/log/auth.log"
    start_position => "beginning"
  }
}

filter {
    if (1 == 2) {
    }
}

output {
  tcp {
    port => 9443
    host => "localhost"
    ssl_enable => true
    ssl_cacert => "/home/sf/thangld13/244cert/ca.crt"
    ssl_key => "/home/sf/thangld13/244cert/client.key"
    ssl_cert => "/home/sf/thangld13/244cert/client.crt"
    #ssl_verify => true
  }
}

Receiving logstash:

input {
        tcp {
                port => 9443
                mode => "server"
                ssl_enable => true
                ssl_cert => "/home/sf/thangld13/244cert/server.crt"
                ssl_cacert => "/home/sf/thangld13/244cert/ca.crt"
                ssl_key => "/home/sf/thangld13/244cert/server.key"
                #ssl_verify => true
        }
}

filter {
    if (1 == 2) {
    }
}

output {
    stdout {
        codec => rubydebug
    }
}

Receiving logstash output:

{
      "@version" => "1",
          "host" => "127.0.0.1",
    "@timestamp" => 2019-05-09T08:26:47.268Z,
       "message" => "{
**A LOT OF EVENT**...
...{\"@version\":\"1\",\"host\":\"siem\",\"path\":\"/var/log/auth.log\",\"@timestamp\":\"2019-05-09T07:39:08.022Z\",\"message\":\"Sep 20 02:00:15 siem sshd[2167]: channel 5: open failed: connect failed:\"}{\"@version\":\"1\",\"host\":\"siem\",\"path\":\"/var/log/auth.log\",\"@timestamp\":\"2019-05-09T07:39:08.022Z\",\"message\":\"Sep 20 02:00:16 siem sshd[2167]: channel 5: open failed: connect failed:\"}{\"@version\":\"1\",\"host\":\"siem\",\"path\":\"/var/log/auth.log\",\"@timestamp\":\"2019-05-09T07:39:08.022Z\",\"message\":\"Sep 20 02:00:19 siem sshd[2167]: channel 5: open failed: connect failed:\"}{\"@version\":\"1\",\"host\":\"siem\",\"path\":\"/var/log/auth.log\",\"@timestamp\":\"2019-05-09T07:39:08.022Z\",\"message\":\"Sep 20 02:00:19 siem sshd[2167]: channel 6: open failed: connect failed:\"}{\"@version\":\"1\",\"host\":\"siem\",\"path\":\"/var/log/auth.log\",\"@timestamp\":\"2019-05-09T07:39:08.023Z\",\"message\":\"Sep 20 02:00:23 siem sshd[2167]: channel 5: open failed: connect failed:\"}{\"@version\":\"1\",\"host\":\"siem\",\"path\":\"/var/log/auth.log\",\"@timestamp\":\"2019-05-09T07:39:08.023Z\",\"message\":\"Sep 20 02:00:23 siem sshd[2167]: channel 6: open failed: connect failed:\"}{\"@version\":\"1\",\"host\":\"siem\",\"path\":\"/var/log/auth.log\",\"@timestamp\":\"2019-05-09T07:39:08.023Z\",\"message\":\"Sep 20 02:00:27 siem sshd[2167]: channel 5: open failed: connect failed:\"}{\"@version\":\"1\",\"host\":\"siem\",\"path\":\"/var/log/auth.log\",\"@timestamp\":\"2019-05-09T07:39:08.023Z\",\"message\":\"Sep 20 02:00:27 siem sshd[2167]: channel 6: open failed: connect failed:\"}{\"@version\":\"1\",\"host\":\"siem\",\"path\":\"/var/log/auth.log\",\"@timestamp\":\"2019-05-09T07:39:08.023Z\",\"message\":\"Sep 20 02:00:30 siem sshd[2167]: channel 5: open failed: connect failed:\"}{\"@version\":\"1\",\"host\":\"siem\",\"path\":\"/var/log/auth.log\",\"@timestamp\":\"2019-05-09T07:39:08.023Z\",\"message\":\"Sep 20 02:00:32 siem sshd[2167]: channel 5: open failed: connect failed:\"}",
          "port" => 57976
}

thangld · May 9, 2019, 10:20am

Problem solved!
I finally manage to solve my problem by adding this line to both TCP input and TCP output:
codec => "json_lines"

system · June 6, 2019, 10:20am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Logstash TCP/SSL \| General OpenSslEngine problem Logstash	2	1483	January 9, 2019
SSL Error when using SSL with TCP output plugin Logstash	2	1116	June 18, 2019
Configure tcp input to accept syslog over ssl Logstash	6	1245	July 23, 2020
[Logstash] SSL TCP input certificate issue Logstash	2	409	April 20, 2023
Logstash TCP SSL Certificate Error Logstash	3	1494	July 6, 2017

Logstash TCP output to logstash TCP input with SSL error

Related topics