Read error looking for ack: read tcp x.x.x.x:8080 i/o timeout

Elastic Stack Logstash
1

I'm getting this error message on Logstash-Forwarder. Since the logs are not sent to Logstash-Server. I have cross verified the Network connection. There is no blocking / Disturbance in Network [ Firewall, Server iptables and so on] as both Forwarder and Server in LAN network.

I tried with telnet also, its success on the connection.

Please someone help me on this. Still I'm wondering what is the reason behind this error. Almost 4 Months gone, I couldn't find a solution for this. Also post in some community but don't get an solution for my problem. Finally come to this

Error Message : Read error looking for ack: read tcp 10.11.12.13:1223 i/o timeout

2

Are you seeing that error on the server? It sounds like a firewall issue. Double-check your firewall rules and ensure you are allowing all traffic, in particular you have an appropriate rule to allow related/established traffic. It looks like you are accepting the intial SYN packet from the client but are dropping/blocking it's ACK

1 Like
3

@Joshua_Rich Thanks for your reply.

I'm seeing the error on the Logstatsh-forwarder log file. Yes, I doubled check the firewall config in my server. its not enabled. Also, My testing servers are in LAN network.

To confirm whether there may be any issue with LAN connection, I have setup logstash and logstash-forwarder on same host machine. Still same error occurring. Any idea ? .

4

It would be good to see your Logstash and Logstash forwarder configuration files. Also note, you can still have firewall rules on your local interface, nothing wrong with that, so I'd double check the firewall rules by dumping them with sudo iptables -S.

5

This time, I have setup both logstash server and forwarder in same server. Though the problem is there.

this is my iptables rule that is running on my server.

[root@test ~]# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A FORWARD -d 172.17.0.4/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 22 -j ACCEPT
-A FORWARD -d 172.17.0.4/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 80 -j ACCEPT
-A FORWARD -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 8080 -j ACCEPT
-A FORWARD -d 172.17.0.2/32 ! -i docker0 -o docker0 -p tcp -m tcp --dport 443 -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
[root@test ~]#

6

This is my Logstash-Server and Forwarder config

Forwarder config :
{
"network": {
"servers": [ "logstash.example.com:1223" ],
"timeout": 15,
"ssl ca": "/etc/pki/tls/certs/ca-hointeractive-bundle.crt",
"ssl certificate": "/etc/pki/tls/certs/star.example.com.crt"
},
"files": [
{
"paths": [ "/var/log/nginx/test.example.com/ssl-access.log"
],
"fields": { "type": "nginx" }
}
]
}

Server Config:

input {
lumberjack {
port => 1223
type => "logs"
ssl_certificate => "/etc/pki/tls/certs/star.example.com.crt"
ssl_key => "/etc/pki/tls/private/star.example.com.key"
}
}
filter {
if [type] == "nginx" {
grok {
match => { "message" => "%{NGINXACCESS}" }
}
}
}
output {
elasticsearch {
host => "localhost"
}

stdout { codec => rubydebug }
}

7

That looks sane, except your lumberjack port is now 1223 as opposed to 8080? That's probably a good thing, though I'd recommend using really high port number, somewhere between 50000 and 65000 as low ports like 1223 and 8080 are usually registered for a specific use.

Can you take a look and read through this issue and see if any of it applies to you? Particularly the subtle Elasticsearch configuration file changes and the note about file descriptor limits.

8

Thanks for your comment. I will look into the link that you shared.

Not sure, the port may be a problem here. Anyway, I will give try on that option too.