# Docker Filebeat can't connect to Docker ELK's http://logstash:5044 but can connect to http://logstash:9600 **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [August 29, 2018, 3:36pm UTC](https://discuss.elastic.co/t/docker-filebeat-cant-connect-to-docker-elks-http-logstash-5044-but-can-connect-to-http-logstash-9600/146556 "2018-08-29T15:36:56Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![shirakaba](https://avatars.discourse-cdn.com/v4/letter/s/e9c0ed/32.png) [@shirakaba](https://discuss.elastic.co/u/shirakaba)\ **Post date:** [August 29, 2018, 3:36pm UTC](https://discuss.elastic.co/t/docker-filebeat-cant-connect-to-docker-elks-http-logstash-5044-but-can-connect-to-http-logstash-9600/146556/1 "2018-08-29T15:36:57Z")
This is a continuation of my thread from yesterday: [Docker Filebeat can’t connect to Docker bridge network despite ELK being able to](https://discuss.elastic.co/t/docker-filebeat-cant-connect-to-docker-bridge-network-despite-elk-being-able-to/146408/3). Now I'm trying to send logs from Filebeat to Logstash, rather than directly to ElasticSearch. The error log from Filebeat: ```auto |filebeat_1 | 2018-08-29T15:14:22.786Z|WARN|transport/tcp.go:36|DNS lookup failure "http://logstash:5044": lookup http://logstash:5044: no such host| |---|---|---|---| |filebeat_1 | 2018-08-29T15:14:23.788Z|ERROR|pipeline/output.go:74|Failed to connect: lookup http://logstash:5044: no such host| ``` Here are the relevant parts of my `docker-compose.yml`: ```auto version: '2' services: logstash: build: context: logstash/ args: ELK_VERSION: $ELK_VERSION volumes: - ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro - ./logstash/pipeline:/usr/share/logstash/pipeline:ro ports: - "127.0.0.1:5000:5000" - "127.0.0.1:5044:5044" environment: LS_JAVA_OPTS: "-Xmx2g -Xms2g" networks: - elk depends_on: - elasticsearch networks: elk: external: true ``` ... Where the network elk is externally created via: ```auto sudo docker network create --driver=bridge elk ``` Here's my `logstash.conf`: ```auto input { tcp { port => 5000 } beats { host => logstash port => 5044 codec => json } } output { elasticsearch { hosts => "elasticsearch:9200" } } ``` And `logstash.yml`: ```auto ## Note: issue persists regardless of which of these http.host values I use: # http.host: "0.0.0.0" # http.host: "logstash" http.host: "127.0.0.1" ``` Here's my `filebeat.yml`: ```auto output: logstash: enabled: true hosts: - http://logstash:5044 setup.kibana: host: http://kibana:5601 # inputs removed for brevity ``` logstash logs this on startup: ```auto logstash_1 | [2018-08-29T15:09:06,121][INFO][logstash.inputs.beats] Beats inputs: Starting input listener {:address=>"logstash:5044"} logstash_1 | [2018-08-29T15:09:06,203][INFO][logstash.pipeline] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#"} logstash_1 | [2018-08-29T15:09:06,738][INFO][org.logstash.beats.Server] Starting server on port: 5044 ``` When I perform `curl` requests from inside the Filebeat container, the following happens: ```auto # Before the ELK stack is running bash-4.2$ curl http://logstash:5044 curl: (6) Could not resolve host: logstash; Unknown error # After the ELK stack is running bash-4.2$ curl http://logstash:5044 curl: (56) Recv failure: Connection reset by peer ``` Although none of the requests made automatically upon scraping files result in any log messages appearing in logstash, that second manual `curl` results in the following error (bearing in mind it's an empty request): ```auto logstash_1 | [2018-08-29T15:31:12,132][INFO][org.logstash.beats.BeatsHandler] [local: 172.18.0.5:5044, remote: 172.18.0.2:60002] Handling exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 69 logstash_1 | [2018-08-29T15:31:12,133][WARN][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception. logstash_1 | io.netty.handler.codec.DecoderException: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 69 logstash_1 | at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[logstash-input-tcp-5.0.9.jar:?] logstash_1 | at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[logstash-input-tcp-5.0.9.jar:?] logstash_1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[logstash-input-tcp-5.0.9.jar:?] logstash_1 | at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) ~[logstash-input-tcp-5.0.9.jar:?] logstash_1 | at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353) ~[logstash-input-tcp-5.0.9.jar:?] logstash_1 | at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[logstash-input-tcp-5.0.9.jar:?] logstash_1 | at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [logstash-input-tcp-5.0.9.jar:?] logstash_1 | at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [logstash-input-tcp-5.0.9.jar:?] logstash_1 | at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171] logstash_1 | Caused by: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 69 logstash_1 | at org.logstash.beats.BeatsParser.decode(BeatsParser.java:92) ~[logstash-input-beats-5.0.16.jar:?] logstash_1 | at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[logstash-input-tcp-5.0.9.jar:?] logstash_1 | at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[logstash-input-tcp-5.0.9.jar:?] logstash_1 | ... 8 more ``` Of note, the following curl _does_ work from inside the Filebeat container: ```auto bash-4.2$ curl http://logstash:9600 {"host":"25b903c79df2","version":"6.3.2","http_address":"0.0.0.0:9600","id":"33dd05c8-a197-40f7-8e33-05ab432595a3","name":"25b903c79df2","build_date":"2018-07-20T06:22:39Z","build_sha":"81d8e503909c00058cd0277809965624ee9dfb80","build_snapshot":false} ``` Any idea what I'm doing wrong here? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [August 29, 2018, 6:01pm UTC](https://discuss.elastic.co/t/docker-filebeat-cant-connect-to-docker-elks-http-logstash-5044-but-can-connect-to-http-logstash-9600/146556/2 "2018-08-29T18:01:10Z")
Is filebeat running before logstash? Logstash startup can take 1 or 2 seconds, but filebeat might be ready to send logs within a few milliseconds. In this case port 5044 is not available yet. Filebeat will retry in this case. Filebeat uses exponential backoff, and will therefore retry after 1s, 2s, 4s, 8s, 16s, ... (up to 60s by default). If error is only on startup, it is because logstash is not up yet. If error does persist, you have indeed a problem. filebeat-\>logstash is not http. You can use `ping`, `nc` or plain old `telnet` to check connection setup only. The exception in LS is expected, because curl will send some bytes on port 5044, that Logstash does not understand. At least you've been able to connect. Using curl doesn't really cause any harm (the exception can be ignored). It's quite common to also configure some health checks like this: [https://github.com/elastic/beats/blob/master/testing/environments/snapshot.yml#L20](https://github.com/elastic/beats/blob/master/testing/environments/snapshot.yml#L20) And make sure a container startup depends on other services being up an healthy: > ---
**Author:** ![shirakaba](https://avatars.discourse-cdn.com/v4/letter/s/e9c0ed/32.png) [@shirakaba](https://discuss.elastic.co/u/shirakaba)\ **Post date:** [August 29, 2018, 6:12pm UTC](https://discuss.elastic.co/t/docker-filebeat-cant-connect-to-docker-elks-http-logstash-5044-but-can-connect-to-http-logstash-9600/146556/3 "2018-08-29T18:12:08Z")
> [@steffens](#): > > Is filebeat running before logstash? I always ensure that Logstash sets up first (except for that one quote where I compared `curl` results before and after the ELK stack starting up). I've been testing this for several hours and haven't had any successful communication from Filebeat to Logstash yet. > [@steffens](#): > > filebeat-\>logstash is not http. You can use `ping` , `nc` or plain old `telnet` to check connection setup only. Thank you for clarifying this. > [@steffens](#): > > It's quite common to also configure some health checks like this: This is attractive; in my ELK container, both Logstash and Kibana specify `depends_on: ['elasticsearch']`. However, I've set up Filebeat in a separate compose script container altogether, so I presume I can't use this feature? > [@steffens](#): > > If error does persist, you have indeed a problem. In any case, I don't think it's a race condition based on Logstash starting up; my connections fail indefinitely. What should I check? ---
**Author:** ![shirakaba](https://avatars.discourse-cdn.com/v4/letter/s/e9c0ed/32.png) [@shirakaba](https://discuss.elastic.co/u/shirakaba)\ **Post date:** [August 29, 2018, 6:31pm UTC](https://discuss.elastic.co/t/docker-filebeat-cant-connect-to-docker-elks-http-logstash-5044-but-can-connect-to-http-logstash-9600/146556/4 "2018-08-29T18:31:09Z")
No more information garnered from using `ping`: ```auto bash-4.2$ ping http://logstash:5044 ping: http://logstash:5044: Name or service not known # nc and telnet not installed on this container ``` ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [August 29, 2018, 7:16pm UTC](https://discuss.elastic.co/t/docker-filebeat-cant-connect-to-docker-elks-http-logstash-5044-but-can-connect-to-http-logstash-9600/146556/5 "2018-08-29T19:16:24Z")
the ping command must be `$ ping logstash`. Ping doesn't know anything about HTTP or ports. Even if you have a custom container, this one is based on a real OS, no? I think you can install missing tools via apt-get or yum even from within the container. > I've been testing this for several hours and haven't had any successful communication from Filebeat to Logstash yet. This leaves me a little confused. curl was able to connect to the logstash host, correct? From where did you run both curl commands? These use of IP 127.0.0.1 makes me wonder if you accidentally bind the logstash ports to the internal container only, such that the ports are not available to other containers. Does it work if you remove the IP in your config? ---
**Author:** ![shirakaba](https://avatars.discourse-cdn.com/v4/letter/s/e9c0ed/32.png) [@shirakaba](https://discuss.elastic.co/u/shirakaba)\ **Post date:** [August 29, 2018, 7:30pm UTC](https://discuss.elastic.co/t/docker-filebeat-cant-connect-to-docker-elks-http-logstash-5044-but-can-connect-to-http-logstash-9600/146556/6 "2018-08-29T19:30:45Z")
> [@steffens](#): > > the ping command must be `$ ping logstash` . Ping doesn't know anything about HTTP or ports. Got it: > [@steffens](#): > > I think you can install missing tools via apt-get or yum even from within the container. `apt-get` worked, yes – sorry, still getting used to Docker, to be honest. ```auto bash-4.2$ ping logstash PING logstash (172.18.0.4) 56(84) bytes of data. 64 bytes from docker-elk_logstash_1.elk (172.18.0.4): icmp_seq=1 ttl=64 time=0.842 ms ``` > [@steffens](#): > > This leaves me a little confused. curl was able to connect to the logstash host, correct? From where did you run both curl commands? From inside the Filebeat container. See the next bit: > [@steffens](#): > > filebeat-\>logstash is not http. This was the key. My `filebeat.yml` was: ```auto output: logstash: enabled: true hosts: - http://logstash:5044 ``` I needed to remove the `http://` protocol: ```auto output: logstash: enabled: true hosts: - logstash:5044 ``` ... And that's it! This explains how `curl` was working to port `9600` but not to `5044`; `9600` is HTTP. Thank you again! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 26, 2018, 7:30pm UTC](https://discuss.elastic.co/t/docker-filebeat-cant-connect-to-docker-elks-http-logstash-5044-but-can-connect-to-http-logstash-9600/146556/7 "2018-09-26T19:30:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.