Logstash not receiving Docker syslog input

[jjarzyns]

I am facing an issue with the ELK stack and docker syslog. I used tcpdump and confirmed that log entries are being sent to the port exposed by my Logstash container but for some reason they don't show up in the container's logs and in Elastic.

I am at a loss for what else to check.

logstash.conf :

input {
	tcp {
		port => 5000
		type => syslog
	}
}

output {
	elasticsearch {
		hosts => "elasticsearch:9200"
		user => "elastic"
		password => "changeme"
	}
	stdout { codec => rubydebug }
}

docker-compose.yml:

(...)
logging:
    driver: syslog
    options:
        syslog-address: "tcp://localhost:5000"
        tag: "api-core"
(...)

After $ sudo tcpdump -A -i any dst port 5000 I can see the log entries appearing.

After docker service logs -f default_logstash it stays at:

dev_logstash.1.to4tzbuaaxlq@dev-1    | [2019-08-26T10:39:38,015][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

[Badger]

A tcp input reads newline separated log entries. If there are no newlines then it will just keep buffering data until it sees one.

[jjarzyns]

I have been able to narrow it down to the difference between docker-compose up and docker stack deploy. But I still don't understand how it happens and need to solve it.

I forked the docker-elk repository that I used so it is easy to outline the steps leading to the behaviour I am seeing:

$ git clone https://github.com/jjarzynski/docker-elk-fork.git
$ cd docker-elk-fork
$ docker-compose up -d
$ telnet localhost 5000
Trying ::1...
Connected to localhost.
Escape character is '^]'.
foo
$ docker logs dockerelkfork_logstash_1
[2019-08-27T08:26:25,147][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print- 
   1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is 
deprecated
{
   "message" => "foo\r",
   "type" => "syslog",
   "host" => "gateway",
   "@timestamp" => 2019-08-27T08:26:38.553Z,
   "port" => 33350,
   "@version" => "1"
}

So with docker-compose what I send over telnet ends up in Logstash, which is not the case with docker stack deploy:

$ docker stack deploy -c docker-stack.yml elk
$ docker ps
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS              PORTS                NAMES
015eef7b17c2        docker.elastic.co/kibana/kibana:7.2.1                 "/usr/local/bin/kiba…"   4 seconds ago       Up 1 second         5601/tcp             elk_kibana.1.scfua5s15r35mqxf37stonupi
1ac91a04bdae        docker.elastic.co/logstash/logstash:7.2.1             "/usr/local/bin/dock…"   7 seconds ago       Up 6 seconds        5044/tcp, 9600/tcp   elk_logstash.1.ku0icfquew98gk6kc7awyd3pj
1dcb4f95fd0a        docker.elastic.co/elasticsearch/elasticsearch:7.2.1   "/usr/local/bin/dock…"   10 seconds ago      Up 8 seconds        9200/tcp, 9300/tcp   elk_elasticsearch.1.1awq5obgxpbix8knhx65yed3d
$ telnet localhost 5000
Trying ::1...
Connected to localhost.
Escape character is '^]'.
hey
$ docker service logs -f elk_logstash
elk_logstash.1.aaxnhluu7w3v@dev-1    | [2019-08-27T08:31:39,803][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}