Logstash Exec Input Plugin throws OutofMemory Error

Elastic Stack Logstash
1

Hi,

We are running a program to read some events off our API and send them to Elasticsearch. Since http poller plugin does not support dynamic parameters (in our case timestamp when last poll was done), we use a custom shell script which keeps track of last event pulled and calls API for all events since that event. The shell script is run using exec input plugin. Our logstash version is 6.3.2.

The exec plugin worked correctly for some time but lately we see errors like

[ERROR][logstash.inputs.exec ] Error while running command {:command=>"/usr/local/scripts/snort_alerts 2> /dev/null", :e=>#<Errno::ENOMEM: Cannot allocate memory - /usr/local/scripts/snort_alerts 2> /dev/null>, :backtrace=>["org/jruby/RubyIO.java:3835:in popen'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-exec-3.3.1/lib/logstash/inputs/exec.rb:97:in run_command'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-exec-3.3.1/lib/logstash/inputs/exec.rb:71:in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-exec-3.3.1/lib/logstash/inputs/exec.rb:52:in run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:512:in inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:505:in block in start_input'"]}

It appears like exec cannot spawn the child process. Rest of logstash/system works correctly and all other pipelines continue processing.

The system has 16GB RAM and Logstash has -Xmx and -Xms set to 8g. The snapshot of free -m when we see the errors is

$ free -m
              total        used        free      shared  buff/cache   available
Mem:          15508        8700        5389         167        1418        6266
Swap:          1906         255        1651

The plugin configuration is

input {
	exec {
		command => "/usr/local/scripts/snort_alerts 2> /dev/null"
		interval => 60
		codec => "json"
		type => "snort_alert"
	}
}

Restarting logstash temporarily corrects the problem but it reappears after some time.

The code for plugin uses IO.popen to spawn the subprocess but this answer on Stackoverflow says using IO.popen calls the fork system call which duplicates entire memory space of parent process (in this case logstash). Could this be the problem?

1 Like
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.