Cannot make Logstash read syslog

AJ_NOURI · May 4, 2018, 9:10pm

Cannot make logstash receive syslog 514 traffic.
ELK run successfully and listens to port 514:

# docker-compose ps
          Name                        Command               State                                  Ports                                 
----------------------------------------------------------------------------------------------------------------------------------------
logstash_elasticsearch_1   /docker-entrypoint.sh elas ...   Up      0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp                       
logstash_kibana_1          /docker-entrypoint.sh kibana     Up      0.0.0.0:5601->5601/tcp                                               
logstash_mylogstash_1      /docker-entrypoint.sh bash       Up      0.0.0.0:5000->5000/tcp, 0.0.0.0:514->514/tcp, 0.0.0.0:8080->8080/tcp

I have tried both plugins syslog and tcp/udp listening on port 514:

1- Syslog plugin:

input {
  syslog { }
}

filter {
}

output {
    stdout {
        codec => rubydebug
    }
}

Result From inside logstash container ==> Logstash do not react to syslog traffic:

21:03:29.310 [[main]-pipeline-manager] INFO logstash.pipeline - Pipeline main started
21:03:29.344 [Ruby-0-Thread-12: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-3.4.1/lib/logstash/inputs/syslog.rb:109] INFO logstash.inputs.syslog - Starting syslog udp listener {:address=>"0.0.0.0:514"}
21:03:29.357 [Ruby-0-Thread-14: /usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-syslog-3.4.1/lib/logstash/inputs/syslog.rb:113] INFO logstash.inputs.syslog - Starting syslog tcp listener {:address=>"0.0.0.0:514"}
21:03:29.533 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=>9600}

2- TCP/UDP plugin:

input{
      tcp {
        port => 514
        type => syslog
      }
      udp {
        port => 514
        type => syslog
      }
    }

filter {
}
    

output {
    stdout {
        codec => rubydebug
    }
}

Result ==> From inside logstash container: looks like something missing?

16:27:55.840 [[main]<udp] WARN logstash.inputs.udp - UDP listener died {:exception=>#<NameError: uninitialized constant LogStash::Inputs::Udp::IPAddr>, :backtrace=>["org/jruby/RubyModule.java:2746:in `const_missing'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.3.1/lib/logstash/inputs/udp.rb:87:in `udp_listener'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.3.1/lib/logstash/inputs/udp.rb:57:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:470:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:463:in `start_input'"]}

Any hint guys, I am stuck here!!

atira · May 4, 2018, 10:11pm

I wonder if it would work if you sent syslog data to a non-priviliged port (>1023).
Just an idea. Though if logstash is running as root, it should be no problem.

Btw. did you try to run a tcp and a udp listener on the same port at the same time?

AJ_NOURI · May 5, 2018, 1:24am

Hi @atira, thanks for your reply.

Removed tcp and left only udp 514 and then with port > 1024: ==> same result

00:50:41.114 [[main]<udp] WARN logstash.inputs.udp - UDP listener died {:exception=>#<NameError: uninitialized constant LogStash::Inputs::Udp::IPAddr>, :backtrace=>["org/jruby/RubyModule.java:2746:in `const_missing'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.3.1/lib/logstash/inputs/udp.rb:87:in `udp_listener'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.3.1/lib/logstash/inputs/udp.rb:57:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:470:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:463:in `start_input'"]}

input{
  udp {
    # port => 514
    port => 5000
    type => syslog
  }
}

filter {
}
    

output {
    stdout {
        codec => rubydebug
    }
}

Should be doing something fundamentally wrong

The plugins looks there:

root@e3a65c2bb61c:/# logstash-plugin list
...
logstash-input-syslog
logstash-input-tcp
logstash-input-udp
root@e3a65c2bb61c:/#

atira · May 5, 2018, 8:11am

type uses a string value, so it needs to be put in quotes

type => "syslog"

AJ_NOURI · May 6, 2018, 4:19am

Put in double quotes, still the same error:

04:19:19.999 [[main]<udp] INFO logstash.inputs.udp - Starting UDP listener {:address=>"0.0.0.0:5000"}
04:19:20.000 [[main]<udp] WARN logstash.inputs.udp - UDP listener died {:exception=>#<NameError: uninitialized constant LogStash::Inputs::Udp::IPAddr>, :backtrace=>["org/jruby/RubyModule.java:2746:in `const_missing'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.3.1/lib/logstash/inputs/udp.rb:87:in `udp_listener'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-udp-3.3.1/lib/logstash/inputs/udp.rb:57:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:470:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:463:in `start_input'"]}

atira · May 6, 2018, 9:05am

Entering to a new field for me, never actually tried to debug open source code, but I've taken a look at it.

You are using version 3.3.1 of the udp plugin, which has the following issue according to the github resource that has been fixed in the current 3.3.2 version:

Fix missing require for the ipaddr library.

And your error seems to be indicating the same problem.

Long story short: update your UDP plugin to the current version.

AJ_NOURI · May 7, 2018, 1:17am

Updated the udp plugin:

root@9a9585c43cfc:/# logstash-plugin update
Updating ...
Updated logstash-codec-netflow 3.11.4 to 3.13.2
Updated logstash-filter-mutate 3.3.1 to 3.3.2
Updated logstash-input-beats 3.1.30 to 3.1.31
Updated logstash-input-file 4.0.5 to 4.1.2
Updated logstash-input-http 3.0.10 to 3.1.0
Updated logstash-input-s3 3.3.2 to 3.3.3
Updated logstash-input-udp 3.3.1 to 3.3.2
Updated logstash-output-s3 4.1.1 to 4.1.2

Looks like the plugin doesn't listen to the socket. From a remote machine:

Nothing listening on port 5001:

ajn@~/tmp$ nc -zv X.X.239.154 5001
nc: connect to X.X.239.154 port 5001 (tcp) failed: Connection refused

THis is confirmed by the packet capture:

Eventhough, for example kibana port works:

ajn@~/tmp$ nc -zv X.X.239.154 5601
Connection to X.X.239.154 5601 port [tcp/*] succeeded!
ajn@~/tmp$

No port restriction on the server or in front of it.

eventhough, logstash start shows "Starting UDP listener "

01:00:59.393 [[main]-pipeline-manager] INFO logstash.pipeline - Pipeline main started
01:00:59.424 [[main]<udp] INFO logstash.inputs.udp - Starting UDP listener {:address=>"0.0.0.0:5001"}
01:00:59.492 [[main]<udp] INFO logstash.inputs.udp - UDP listener started {:address=>"0.0.0.0:5001", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
01:00:59.569 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=>9600}

this is the config:

input {
#    syslog {}
  udp {
    port => 5001
    type => "syslog"
  }

}
filter {
}
    
output {
    stdout {
        codec => rubydebug
    }
}

And I used the following docker-compose to start ELK on the server side:

# cat docker-compose.yml 
version: '2'
services:
    elasticsearch:
        image: elasticsearch
        ports:
            - "9200:9200"
            - "9300:9300"
        volumes:
            - ./data:/usr/share/elasticsearch/data
    mylogstash:
        build:
            context: .
            dockerfile: Dockerfile
        volumes:
            - ./config-dir:/mylogstash/config-dir:rw
            - ./data:/mylogstash/data:rw
        ports:
            - "5000:5000"
            - "5001:5001"
            - "514:514"
        tty: true
        command: ["bash"]
        links:
            - elasticsearch
    kibana:
        image: kibana
        ports:
            - "5601:5601"
        links:
            - elasticsearch

And start correctly as expected:

# docker-compose  ps
         Name                     Command                    State                     Ports          
-----------------------------------------------------------------------------------------------------
logstash_elasticsearch_   /docker-entrypoint.sh     Up                        0.0.0.0:9200->9200/tcp, 
1                         elas ...                                            0.0.0.0:9300->9300/tcp  
logstash_kibana_1         /docker-entrypoint.sh     Up                        0.0.0.0:5601->5601/tcp  
                          kibana                                                                      
logstash_mylogstash_1     /docker-entrypoint.sh     Up                        0.0.0.0:5000->5000/tcp, 
                          bash                                                0.0.0.0:5001->5001/tcp, 
                                                                              0.0.0.0:514->514/tcp

AJ_NOURI · May 7, 2018, 11:07pm

Tested with installed version and it works.
Either something wrong with my ELK docker-compsoe config. or with logstash container

Are ELK containers production ready?

rcowart · May 8, 2018, 5:33am

Notice in the output of docker-compose ps you see only TCP ports lists. No UDP. In your docker-compose file you should try:

ports:
    - "5000/tcp:5000/tcp"
    - "5001/udp:5001/udp"
    - "514/udp:514/udp"
    - "514/tcp:514/tcp"

Adjust for whatever ports you end up using.

This is explained here... https://github.com/moby/moby/issues/32958

AJ_NOURI · May 9, 2018, 3:03am

Thanks for the hint @rcowart .
Indeed it was only listening to tcp.

And it accepts the below foramt:

ports:
    - "5000:5000/tcp"

Otherwise it throws an error:

services.mylogstash.ports is invalid: Invalid port "5000/tcp:5000/tcp", should be [[remote_ip:]remote_port[-remote_port]:]port[/protocol]

Now it also listens to udp:

$ docker-compose ps
          Name                         Command               State                    Ports                 
------------------------------------------------------------------------------------------------------------
elksyslog_elasticsearch_1   /docker-entrypoint.sh elas ...   Up      0.0.0.0:9200->9200/tcp,                
                                                                     0.0.0.0:9300->9300/tcp                 
elksyslog_kibana_1          /docker-entrypoint.sh kibana     Up      0.0.0.0:5601->5601/tcp                 
elksyslog_mylogstash_1      /docker-entrypoint.sh bash       Up      0.0.0.0:5000->5000/tcp,                
                                                                     0.0.0.0:5001->5001/udp,                
                                                                     0.0.0.0:514->514/udp,                  
                                                                     0.0.0.0:8080->8080/tcp

Now it works fine!!
Thanks alot @rcowart

system · June 6, 2018, 3:03am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.