INFO: I/O exception

Elastic Stack Logstash
1

This is my setup

IIS Server --> logstash-forwarder --> 2 logstash instances (fqdn certs) --> haproxy --> 2 elasticsearch node

The logstash-forwarder is working and my cluster is also mostly in green.. but out of nowhere for no reason logstash-forwarder seems to stop.

had a look at the logs in the logstash instances and this is what i found

/var/log/logstash/logstash.err

INFO: I/O exception (org.apache.http.NoHttpResponseException) caught when processing request to {}->http://xx.xx.x.123:9200: The target server failed to respond
Jun 16, 2015 9:04:56 AM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://xx.xx.xx.123:9200

http://xx.xx.xx.123:9200 <-- this is my haproxy ip!

This is my logstash output config

output  {
    if "_grokparsefailure" not in [tags]    {
            elasticsearch   {
                    cluster => "elkstackz"
                    host => "xx.xx.xx.123"
                    protocol => "http"
                    user => "xxxx"
                    password => "xxxxx"
            }
    }
    file   {
            tags => "_grokparsefailure_sysloginput"
            path => "/var/log/grokparsefailure"
    }
}

Is this because the 2 elastic nodes are having bad I/O? I am not using an SSD. Anyways i guess its failing from the haproxy?

regards,

Ismail

2

I'd be checking your HAproxy logs.

Also note that when using protocol => "http" for the elasticsearch Logstash output, you can pass an array of Elasticsearch nodes to the host parameter and the output plugin will load balance the bulk requests between them. So you could essentially drop HAproxy out of the pipeline by specifying you Logstash output like:

elasticsearch   {
                    cluster => "elkstackz"
                    host => [ "xx.xx.xx.124", "xx.xx.xx.125" ]
                    protocol => "http"
                    user => "xxxx"
                    password => "xxxxx"
            }

Where xx.xx.xx.124 and xx.xx.xx.125 are you Elasticsearch nodes.

3

Thanks, I will get rid of the HAProxy this way. but anyways i found my issue.

iis w3svc log format sets the timezone to GMT.. so my logs were showing a backdate.. i thought logstash-forwarder was having some issue.

https://support.microsoft.com/en-us/kb/271196

i need to change it to microsoft iis log format to be able to change the timezone so it actually starts logging with the proper time..

but i am still curious about the error messages i saw in the logstash.err with INFO: I/O exception. Anyways thanks :slight_smile: This project is fantastic.