IIS Logs

Elastic Stack Logstash
1

Is it possible to have too large of IIS logs for Elasticsearch?

For example, there are at least ten servers sending IIS logs that are about 3GB each per file.

I have noticed today that IIS logs have not been in Kibana for the last few hours.

I do logstash logs | tail -n 50 and see the following ::

2016-05-10 23:47:09 +0000: HTTP parse error, malformed request (): #<Puma::HttpParserError: Invalid HTTP format, parsing fails.>
2016-05-10 23:47:09 +0000: ENV: {"rack.version"=>[1, 3], "rack.errors"=>#<IO:fd 2>, "rack.multithread"=>true, "rack.multiprocess"=>false, "rack.run_once"=>false, "SCRIPT_NAME"=>"", "CONTENT_TYPE"=>"text/plain", "QUERY_STRING"=>"", "SERVER_PROTOCOL"=>"HTTP/1.1", "SERVER_SOFTWARE"=>"2.11.3", "GATEWAY_INTERFACE"=>"CGI/1.2"}

2016-05-10 23:47:09 +0000: HTTP parse error, malformed request (): #<Puma::HttpParserError: Invalid HTTP format, parsing fails.>
2016-05-10 23:47:09 +0000: ENV: {"rack.version"=>[1, 3], "rack.errors"=>#<IO:fd 2>, "rack.multithread"=>true, "rack.multiprocess"=>false, "rack.run_once"=>false, "SCRIPT_NAME"=>"", "CONTENT_TYPE"=>"text/plain", "QUERY_STRING"=>"", "SERVER_PROTOCOL"=>"HTTP/1.1", "SERVER_SOFTWARE"=>"2.11.3", "GATEWAY_INTERFACE"=>"CGI/1.2"}

2016-05-10 23:48:09 +0000: HTTP parse error, malformed request (): #<Puma::HttpParserError: Invalid HTTP format, parsing fails.>
2016-05-10 23:48:09 +0000: ENV: {"rack.version"=>[1, 3], "rack.errors"=>#<IO:fd 2>, "rack.multithread"=>true, "rack.multiprocess"=>false, "rack.run_once"=>false, "SCRIPT_NAME"=>"", "CONTENT_TYPE"=>"text/plain", "QUERY_STRING"=>"", "SERVER_PROTOCOL"=>"HTTP/1.1", "SERVER_SOFTWARE"=>"2.11.3", "GATEWAY_INTERFACE"=>"CGI/1.2"}

2016-05-10 23:48:09 +0000: HTTP parse error, malformed request (): #<Puma::HttpParserError: Invalid HTTP format, parsing fails.>
2016-05-10 23:48:09 +0000: ENV: {"rack.version"=>[1, 3], "rack.errors"=>#<IO:fd 2>, "rack.multithread"=>true, "rack.multiprocess"=>false, "rack.run_once"=>false, "SCRIPT_NAME"=>"", "CONTENT_TYPE"=>"text/plain", "QUERY_STRING"=>"", "SERVER_PROTOCOL"=>"HTTP/1.1", "SERVER_SOFTWARE"=>"2.11.3", "GATEWAY_INTERFACE"=>"CGI/1.2"}

2016-05-10 23:49:09 +0000: HTTP parse error, malformed request (): #<Puma::HttpParserError: Invalid HTTP format, parsing fails.>
2016-05-10 23:49:09 +0000: ENV: {"rack.version"=>[1, 3], "rack.errors"=>#<IO:fd 2>, "rack.multithread"=>true, "rack.multiprocess"=>false, "rack.run_once"=>false, "SCRIPT_NAME"=>"", "CONTENT_TYPE"=>"text/plain", "QUERY_STRING"=>"", "SERVER_PROTOCOL"=>"HTTP/1.1", "SERVER_SOFTWARE"=>"2.11.3", "GATEWAY_INTERFACE"=>"CGI/1.2"}

I am seeing a LOT of these messages lately.

I also try logstash logs | grep "IIS" .........with no results found

2

Also, here is a little bit of information about the ELK stack ::

4 Nodes

Node01 - 4 Cores 28 GB Memory
Node02 - 4 Cores 28 GB Memory
Node03 - 2 Cores 14 GB Memory
Node04 - 2 Cores 14 GB Memory

Node01 and Node02 are "hot" nodes, which only keep logs for 5 days before moving them to the other two nodes, which are "warm" and "cold" nodes

3

That looks like something that is not ES, do you have a load balancer somewhere?