Issue in Logstash after Upgrade

Elastic Stack Logstash
1

I tested upgrade of ELK stack to V5.6.8 from V5.2.2 running on Centos 7 VM . Elasticsearch upgrade & Kibana upgrade Went fine ..But logstash ran into issue details below

testing Logatsh config :
input {
udp {
type => "arista"
port => "514"
host => "x.x.x.x"
}

     tcp {
   ssl_verify => false
   type => "arista"
   port => "514"
   host => "x.x.x.x"
 }
}

output {
if [type] == "arista" {
elasticsearch {
ssl => false
ssl_certificate_verification => false
hosts => ["x.X.X.X:9200"]
index => "ipsla-%{+YYYY.MM.dd}"
}
}
}

Error :
[2018-03-11T18:32:44,041][ERROR][logstash.pipeline ] A plugin had an unrecoverable error. Will restart this plugin.
Plugin: <LogStash::Inputs::Tcp ssl_verify=>false, type=>"arista", port=>514, host=>"10.67.10.208", id=>"6b81d8f1ee654ee08ae4c5ac55c7a190737fb508-2", enable_metric=>true, codec=><LogStash::Codecs::Line id=>"line_923f272b-69ab-4156-a39d-5c12be7043aa", enable_metric=>true, charset=>"UTF-8", delimiter=>"\n">, data_timeout=>-1, mode=>"server", proxy_protocol=>false, ssl_enable=>false, ssl_key_passphrase=>>
Error: Permission denied

When i enable only UDP port on the logstash config it gives me another error

[2018-03-11T18:40:23,004][WARN ][logstash.inputs.udp ] UDP listener died {:exception=>#<SocketError: bind: name or service not known>, :backtrace=>["org/jruby/ext/socket/RubyUDPSocket.java:164:in bind'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.2.1/lib/logstash/inputs/udp.rb:95:inudp_listener'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.2.1/lib/logstash/inputs/udp.rb:56:in run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:470:ininputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:463:in `start_input'"]}

What will be the reason for the issue ? ...Please help

2

I found a relevant issue on github, with comments that should provide a path forward:

Ports in range 1 to 1024 are privileged and only root user can listen on it.

Options:

  • run logstash as root (not a good idea)
  • use setcap to grant java permission to use privileged ports
  • use iptables or a proxy to forward port 514 to an unprivileged port.

-- logstash will not open a listening port. · Issue #1587 · elastic/logstash · GitHub

This blog post is specifically about using setcap to set capabilities for Java (although it references Java 7, and Logstash currently requires Java 8, so you may need to adjust it slightly):

3

Hi yaauie ..Appreciate your help ..The logstash is now listening on specific ports on the Config

But for some reason its not accepting logs from the clients ..Below is the test logstash config ..I dont see any logs on ruby debug ..I can see logs coming on ruby debug when i add self hearbeat in the input section ..I believe some issue in Input plugin ..Not sure ..Please advise

####NAGIOS###
input {
udp {
type => "nagios"
port => "1514"
host => "X.X.X.X"
}

 tcp {
   type => "nagios"
   ssl_verify => false
   port => "1514"
   host => "X.X.X.X"
 }
}

filter {

if [type] == "nagios" {

grok {

     match => [
        # IOS
        "message", "%{IP:host_nagios}: Nagios-Log device_id=%{WORD:hostname} rtt=%{NUMBER:RTT} avgSD=%{NUMBER:SDLATENCY} avgDS=%

{NUMBER:DSLATENCY} syslog_sev_level=%{INT:syslog_sev_level} syslog_severity=%{WORD:syslog_severity} hostgroup=%{WORD:hostgroup} %{GR
EEDYDATA:log_message}"
]
add_tag => [ "Nagios" ]
}
}

if "Nagios" in [tags]
{

mutate { convert => { "syslog_sev_level" => "integer" } }
mutate { convert => { "RTT" => "float" } }
mutate { convert => { "SDLATENCY" => "float" } }
mutate { convert => { "DSLATENCY" => "float" } }

   }

}

output {
if [type] == "nagios" {
elasticsearch {
hosts => ["X.X.X.X:9200"]
index => "nagios-%{+YYYY.MM.dd}"

user => "XXXXX"

password => "XXXXXX"

    }

}

For troubleshoot debug use

file{
path => "/tmp/nagios_1514.txt"
codec => rubydebug
}
}

4

In future, adding markdown code blocks around blocks of code (a line with three tildes like ~~~ will begin or end a block) makes reading it a lot easier.

What is sending the logs? Is it configured to send to port 1514, and has it been restarted recently to pick up the change? (When you opened this ticket, things were pointed at port 514).

5

We understood the problem with using 514 ..Was testing one of the logstash config hearing on port 1514 which is higher port that 1024 ..When pointing logstash config to port 1514 the instance is up as below

[root@elkuat01 tmp]#
[root@elkuat01 tmp]# netstat -an | grep 1514
tcp6 0 0 10.67.10.208:1514 :::* LISTEN
udp6 0 0 10.67.10.208:1514 :::*

But more some reason the logs is not been read to Elasticsearch & also not seen in rubydebug.But in tcpdump on the VM shows traffic from the client on port 1514 ..Not sure what is happening ..

6

Synopsis: Logstash is apparently listening on the configured port (in this case 1514), but is apparently not receiving any connections.

We can manually send a message to Logstash with nc to validate that it is really listening (replace the 127.0.0.1 with Logstash's external interface IP when testing from another machine):

echo "test-tcp" | nc -w0 127.0.0.1 1514

echo "test-udp" | nc -w0 -u 127.0.0.1 1514

If you see these messages, then Logstash is in fact bound to the ports for both tcp and udp, and the problem lies somewhere between your sender and Logstash (e.g., Logstash inputs being bound to local-only IP address, no route from your other host, firewall blocking port ranges)

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.