Logstash 2.2 OutOfMemoryError

Elastic Stack Logstash
1

We are trying to use Winlogbeats to archive off our Windows logs. To start this, we setup a Linux server running the latest ELK. On one of our three Windows 2012 R2 DC's we setup winlogbeats to ship the logs to the Ubuntu ELK server. It runs for awhile but eventually Logstash crashes. I see that it is out of memory. How much should I set the cap to? This is just one of three DC's and we were hoping to ship other logs to it also? I think we were getting about 10 records/sec sent to it. I find it hard to believe that we overloaded it?

Here is what we are getting in the in logstash.err.1:

Mar 04, 2016 2:45:00 AM org.apache.http.impl.execchain.RetryExec execute
INFO: I/O exception (java.net.SocketException) caught when processing request to {}->http://localhost:9200: Socket closed
Mar 04, 2016 2:45:00 AM org.apache.http.impl.execchain.RetryExec execute
INFO: Retrying request to {}->http://localhost:9200
Error: Your application used more memory than the safety cap of 1G.
Specify -J-Xmx####m to increase it (#### = cap size in MB).
Specify -w for full OutOfMemoryError stack trace
2

Are you using LSF here? Cause you've posted it in that category.

3

Sorry, changed it.

4

That's looks like a Logstash OOM issue not a Winlogbeat issue. We can help you better if you list the specific versions of Winlogbeat, Logstash, Java, and your operating systems. Also provide the configuration files that are being used.

5

Below are my configs. Any help or suggestions are greatly appreciated!

Logstash server:
Ubuntu 14.04
Java - Oracle Java 8
Logstash 2.2
ElasticSearch 2.2.0

Logstash Configs:

02-beats-input.conf

    input {
      beats {
        port => 5044
        ssl => true
        ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
        ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
      }
    }

30-elasticsearch-output.conf

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    sniffing => true
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

10-syslog-filter.conf

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

Beats Client
Windows Server 2012 R2
Winlogbeat.1.1.1

winlogbeat.yml

winlogbeat:
  registry_file: C:/ProgramData/winlogbeat/.winlogbeat.yml

  event_logs:
    - name: Application
      ignore_older: 1h 
    - name: Security
    - name: System
      ignore_older: 1h
  metrics:
    bindaddress: 'localhost:8123'
output:
  logstash:
    hosts: ["logserver.mydomain.com:5044"]
    index: winlogbeat
    tls:
      certificate_authorities: ["/apps/beats/certs/logstash-forwarder.crt"]

shipper:
logging:
  to_files: true
  files:
    path: C:/ProgramData/winlogbeat/Logs
    rotateeverybytes: 10485760 # = 10MB
  level: info