This is a continuation of my thread from yesterday: Docker Filebeat can’t connect to Docker bridge network despite ELK being able to.
Now I'm trying to send logs from Filebeat to Logstash, rather than directly to ElasticSearch. The error log from Filebeat:
|filebeat_1 | 2018-08-29T15:14:22.786Z|WARN|transport/tcp.go:36|DNS lookup failure "http://logstash:5044": lookup http://logstash:5044: no such host|
|---|---|---|---|
|filebeat_1 | 2018-08-29T15:14:23.788Z|ERROR|pipeline/output.go:74|Failed to connect: lookup http://logstash:5044: no such host|
Here are the relevant parts of my docker-compose.yml
:
version: '2'
services:
logstash:
build:
context: logstash/
args:
ELK_VERSION: $ELK_VERSION
volumes:
- ./logstash/config/logstash.yml:/usr/share/logstash/config/logstash.yml:ro
- ./logstash/pipeline:/usr/share/logstash/pipeline:ro
ports:
- "127.0.0.1:5000:5000"
- "127.0.0.1:5044:5044"
environment:
LS_JAVA_OPTS: "-Xmx2g -Xms2g"
networks:
- elk
depends_on:
- elasticsearch
networks:
elk:
external: true
... Where the network elk is externally created via:
sudo docker network create --driver=bridge elk
Here's my logstash.conf
:
input {
tcp {
port => 5000
}
beats {
host => logstash
port => 5044
codec => json
}
}
output {
elasticsearch {
hosts => "elasticsearch:9200"
}
}
And logstash.yml
:
## Note: issue persists regardless of which of these http.host values I use:
# http.host: "0.0.0.0"
# http.host: "logstash"
http.host: "127.0.0.1"
Here's my filebeat.yml
:
output:
logstash:
enabled: true
hosts:
- http://logstash:5044
setup.kibana:
host: http://kibana:5601
# inputs removed for brevity
logstash logs this on startup:
logstash_1 | [2018-08-29T15:09:06,121][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"logstash:5044"}
logstash_1 | [2018-08-29T15:09:06,203][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x21acf88 run>"}
logstash_1 | [2018-08-29T15:09:06,738][INFO ][org.logstash.beats.Server] Starting server on port: 5044
When I perform curl
requests from inside the Filebeat container, the following happens:
# Before the ELK stack is running
bash-4.2$ curl http://logstash:5044
curl: (6) Could not resolve host: logstash; Unknown error
# After the ELK stack is running
bash-4.2$ curl http://logstash:5044
curl: (56) Recv failure: Connection reset by peer
Although none of the requests made automatically upon scraping files result in any log messages appearing in logstash, that second manual curl
results in the following error (bearing in mind it's an empty request):
logstash_1 | [2018-08-29T15:31:12,132][INFO ][org.logstash.beats.BeatsHandler] [local: 172.18.0.5:5044, remote: 172.18.0.2:60002] Handling exception: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 69
logstash_1 | [2018-08-29T15:31:12,133][WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
logstash_1 | io.netty.handler.codec.DecoderException: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 69
logstash_1 | at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[logstash-input-tcp-5.0.9.jar:?]
logstash_1 | at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[logstash-input-tcp-5.0.9.jar:?]
logstash_1 | at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[logstash-input-tcp-5.0.9.jar:?]
logstash_1 | at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) ~[logstash-input-tcp-5.0.9.jar:?]
logstash_1 | at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353) ~[logstash-input-tcp-5.0.9.jar:?]
logstash_1 | at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:66) ~[logstash-input-tcp-5.0.9.jar:?]
logstash_1 | at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [logstash-input-tcp-5.0.9.jar:?]
logstash_1 | at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [logstash-input-tcp-5.0.9.jar:?]
logstash_1 | at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
logstash_1 | Caused by: org.logstash.beats.BeatsParser$InvalidFrameProtocolException: Invalid Frame Type, received: 69
logstash_1 | at org.logstash.beats.BeatsParser.decode(BeatsParser.java:92) ~[logstash-input-beats-5.0.16.jar:?]
logstash_1 | at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[logstash-input-tcp-5.0.9.jar:?]
logstash_1 | at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[logstash-input-tcp-5.0.9.jar:?]
logstash_1 | ... 8 more
Of note, the following curl does work from inside the Filebeat container:
bash-4.2$ curl http://logstash:9600
{"host":"25b903c79df2","version":"6.3.2","http_address":"0.0.0.0:9600","id":"33dd05c8-a197-40f7-8e33-05ab432595a3","name":"25b903c79df2","build_date":"2018-07-20T06:22:39Z","build_sha":"81d8e503909c00058cd0277809965624ee9dfb80","build_snapshot":false}
Any idea what I'm doing wrong here?