Docker Filebeat can't connect to Docker ELK's http://logstash:5044 but can connect to http://logstash:9600

[shirakaba]

This is a continuation of my thread from yesterday: Docker Filebeat can’t connect to Docker bridge network despite ELK being able to.

Now I'm trying to send logs from Filebeat to Logstash, rather than directly to ElasticSearch. The error log from Filebeat:

|filebeat_1  | 2018-08-29T15:14:22.786Z|WARN|transport/tcp.go:36|DNS lookup failure "http://logstash:5044": lookup http://logstash:5044: no such host|
|---|---|---|---|
|filebeat_1  | 2018-08-29T15:14:23.788Z|ERROR|pipeline/output.go:74|Failed to connect: lookup http://logstash:5044: no such host|

Here are the relevant parts of my docker-compose.yml:

version: '2'

services:
  logstash:
    build:
      context: logstash/
      args:
        ELK_VERSION: $ELK_VERSION
    volumes:
      - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
      - ./logstash/pipeline:/usr/share/logstash/pipeline:ro
    ports:
      - "127.0.0.1:5000:5000"
      - "127.0.0.1:5044:5044"
    environment:
      LS_JAVA_OPTS: "-Xmx2g -Xms2g"
    networks:
      - elk
    depends_on:
      - elasticsearch

networks:
  elk:
    external: true

... Where the network elk is externally created via:

sudo docker network create --driver=bridge elk

Here's my logstash.conf:

input {
    tcp {
        port => 5000
    }
    beats {
        host => logstash
        port => 5044
        codec => json
    }
}

output {
	elasticsearch {
		hosts => "elasticsearch:9200"
	}
}

And logstash.yml:

## Note: issue persists regardless of which of these http.host values I use:
# http.host: "0.0.0.0"
# http.host: "logstash"
http.host: "127.0.0.1"

Here's my filebeat.yml:

output:
  logstash:
    enabled: true
    hosts:
      - http://logstash:5044

setup.kibana:
  host: http://kibana:5601

# inputs removed for brevity

logstash logs this on startup:

logstash_1       | [2018-08-29T15:09:06,121][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"logstash:5044"}
logstash_1       | [2018-08-29T15:09:06,203][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x21acf88 run>"}
logstash_1       | [2018-08-29T15:09:06,738][INFO ][org.logstash.beats.Server] Starting server on port: 5044

When I perform curl requests from inside the Filebeat container, the following happens:

# Before the ELK stack is running
bash-4.2$ curl http://logstash:5044
curl: (6) Could not resolve host: logstash; Unknown error

# After the ELK stack is running
bash-4.2$ curl http://logstash:5044
curl: (56) Recv failure: Connection reset by peer

Although none of the requests made automatically upon scraping files result in any log messages appearing in logstash, that second manual curl results in the following error (bearing in mind it's an empty request):

logstash_1       | [2018-08-29T15:31:12,132][INFO ][org.logstash.beats.BeatsHandler] [local: 172.18.0.5:5044, remote: 172.18.0.2:60002] Handling exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 69
logstash_1       | [2018-08-29T15:31:12,133][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
logstash_1       | io.netty.handler.codec.DecoderException: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 69
logstash_1       | 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[logstash-input-tcp-5.0.9.jar:?]
logstash_1       | 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[logstash-input-tcp-5.0.9.jar:?]
logstash_1       | 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[logstash-input-tcp-5.0.9.jar:?]
logstash_1       | 	at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) ~[logstash-input-tcp-5.0.9.jar:?]
logstash_1       | 	at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353) ~[logstash-input-tcp-5.0.9.jar:?]
logstash_1       | 	at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[logstash-input-tcp-5.0.9.jar:?]
logstash_1       | 	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [logstash-input-tcp-5.0.9.jar:?]
logstash_1       | 	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [logstash-input-tcp-5.0.9.jar:?]
logstash_1       | 	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
logstash_1       | Caused by: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 69
logstash_1       | 	at org.logstash.beats.BeatsParser.decode(BeatsParser.java:92) ~[logstash-input-beats-5.0.16.jar:?]
logstash_1       | 	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[logstash-input-tcp-5.0.9.jar:?]
logstash_1       | 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[logstash-input-tcp-5.0.9.jar:?]
logstash_1       | 	... 8 more

Of note, the following curl does work from inside the Filebeat container:

bash-4.2$ curl http://logstash:9600
{"host":"25b903c79df2","version":"6.3.2","http_address":"0.0.0.0:9600","id":"33dd05c8-a197-40f7-8e33-05ab432595a3","name":"25b903c79df2","build_date":"2018-07-20T06:22:39Z","build_sha":"81d8e503909c00058cd0277809965624ee9dfb80","build_snapshot":false}

Any idea what I'm doing wrong here?

[steffens (Steffen Siering)]

Is filebeat running before logstash? Logstash startup can take 1 or 2 seconds, but filebeat might be ready to send logs within a few milliseconds. In this case port 5044 is not available yet. Filebeat will retry in this case. Filebeat uses exponential backoff, and will therefore retry after 1s, 2s, 4s, 8s, 16s, ... (up to 60s by default). If error is only on startup, it is because logstash is not up yet. If error does persist, you have indeed a problem.

filebeat->logstash is not http. You can use ping, nc or plain old telnet to check connection setup only. The exception in LS is expected, because curl will send some bytes on port 5044, that Logstash does not understand. At least you've been able to connect. Using curl doesn't really cause any harm (the exception can be ignored).

It's quite common to also configure some health checks like this: https://github.com/elastic/beats/blob/master/testing/environments/snapshot.yml#L20

And make sure a container startup depends on other services being up an healthy:

elastic/beats/blob/master/testing/environments/local.yml#L24

[shirakaba]

@steffens

Is filebeat running before logstash?

I always ensure that Logstash sets up first (except for that one quote where I compared curl results before and after the ELK stack starting up). I've been testing this for several hours and haven't had any successful communication from Filebeat to Logstash yet.

@steffens

filebeat->logstash is not http. You can use ping , nc or plain old telnet to check connection setup only.

Thank you for clarifying this.

@steffens

It's quite common to also configure some health checks like this:

This is attractive; in my ELK container, both Logstash and Kibana specify depends_on: ['elasticsearch'].

However, I've set up Filebeat in a separate compose script container altogether, so I presume I can't use this feature?

@steffens

If error does persist, you have indeed a problem.

In any case, I don't think it's a race condition based on Logstash starting up; my connections fail indefinitely. What should I check?

[shirakaba]

No more information garnered from using ping:

bash-4.2$ ping http://logstash:5044
ping: http://logstash:5044: Name or service not known
# nc and telnet not installed on this container

[steffens (Steffen Siering)]

the ping command must be $ ping logstash. Ping doesn't know anything about HTTP or ports.

Even if you have a custom container, this one is based on a real OS, no? I think you can install missing tools via apt-get or yum even from within the container.

I've been testing this for several hours and haven't had any successful communication from Filebeat to Logstash yet.

This leaves me a little confused. curl was able to connect to the logstash host, correct? From where did you run both curl commands?

These use of IP 127.0.0.1 makes me wonder if you accidentally bind the logstash ports to the internal container only, such that the ports are not available to other containers. Does it work if you remove the IP in your config?

[shirakaba]

@steffens

the ping command must be $ ping logstash . Ping doesn't know anything about HTTP or ports.

Got it:

@steffens

I think you can install missing tools via apt-get or yum even from within the container.

apt-get worked, yes – sorry, still getting used to Docker, to be honest.

bash-4.2$ ping logstash
PING logstash (172.18.0.4) 56(84) bytes of data.
64 bytes from docker-elk_logstash_1.elk (172.18.0.4): icmp_seq=1 ttl=64 time=0.842 ms

@steffens

This leaves me a little confused. curl was able to connect to the logstash host, correct? From where did you run both curl commands?

From inside the Filebeat container. See the next bit:

@steffens

filebeat->logstash is not http.

This was the key.

My filebeat.yml was:

output:
  logstash:
    enabled: true
    hosts:
      - http://logstash:5044

I needed to remove the http:// protocol:

output:
  logstash:
    enabled: true
    hosts:
      - logstash:5044

... And that's it!

This explains how curl was working to port 9600 but not to 5044; 9600 is HTTP.

Thank you again!