Logstash restarting every 20-30 seconds

Elastic Stack Logstash
1

Log file has the following error
[2018-07-06T07:50:00,389][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

logstash-simple.conf has following code

input { stdin {} }
output {
elasticsearch { hosts => ["ipadd of ES :9200"] }
stdout { codec => rubydebug }
}
Any kind of suggestion is much appreciated.

2

What comes before that in the log?

3

[2018-07-06T07:54:40,440][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2018-07-06T07:55:09,375][INFO ][logstash.configmanagement.bootstrapcheck] Using Elasticsearch as config store {:pipeline_id=>["main"], :poll_interval=>"30000000000ns"}
[2018-07-06T07:55:09,445][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: You must set the password using the "xpack.management.elasticsearch.password" in logstash.yml>, :backtrace=>["/usr/share/logstash/x-pack/lib/config_management/elasticsearch_source.rb:40:in initialize'", "/usr/share/logstash/x-pack/lib/config_management/hooks.rb:42:inafter_bootstrap_checks'", "/usr/share/logstash/logstash-core/lib/logstash/event_dispatcher.rb:34:in block in fire'", "/usr/share/logstash/logstash-core/lib/logstash/event_dispatcher.rb:32:infire'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:294:in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:inrun'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:238:in run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:inrun'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in <main>'"]} [2018-07-06T07:55:09,465][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit [2018-07-06T07:55:36,740][INFO ][logstash.configmanagement.bootstrapcheck] Using Elasticsearch as config store {:pipeline_id=>["main"], :poll_interval=>"30000000000ns"} [2018-07-06T07:55:36,794][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: You must set the password using the "xpack.management.elasticsearch.password" in logstash.yml>, :backtrace=>["/usr/share/logstash/x-pack/lib/config_management/elasticsearch_source.rb:40:ininitialize'", "/usr/share/logstash/x-pack/lib/config_management/hooks.rb:42:in after_bootstrap_checks'", "/usr/share/logstash/logstash-core/lib/logstash/event_dispatcher.rb:34:inblock in fire'", "/usr/share/logstash/logstash-core/lib/logstash/event_dispatcher.rb:32:in fire'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:294:inexecute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:238:inrun'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in run'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in'"]}
[2018-07-06T07:55:36,811][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

4

ArgumentError: You must set the password using the "xpack.management.elasticsearch.password" in logstash.yml

So... set a password or disable the X-Pack features?

5

After disabled the X-pack, i have the same issue, logstash service is restarting every time,

However below are new logs

[2018-07-06T12:27:13,507][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>RuntimeError, :message=>"Could not fetch the configuration, message: Permission denied - /etc/logstash/conf.d/logstash-simple.conf", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/agent.rb:145:in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:89:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:363:in block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:inblock in initialize'"]}
[2018-07-06T12:27:13,877][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

6

Could not fetch the configuration, message: Permission denied - /etc/logstash/conf.d/logstash-simple.conf

This error message is pretty clear, no?

7

Thanks,

I changed the permission in the file, and getting the below logs

[2018-07-06T13:44:40,300][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2018-07-06T13:44:40,694][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
Still logstash is reloading.

8

I assume logstash-simple.conf isn't empty? If you increase Logstash's log level it might give additional clues.

9

Yes logstash-simple.conf has the following configuration.
input { stdin {} }
output {
elasticsearch { hosts => ["ES server:9200"] }
stdout { codec => rubydebug }
}

i have a issue of displaying logs now, its been stopped on Friday and i didn't see any other logs after that. Any suggestion is much appriciated.

10

@magnusbaeck , Thank you so much for your time to help me with this issue.
Issue has been resolved after the change in the syslogs & permissions settings.

11

Hi.. I am installing Logstash.. I am getting the below error :

C:\Users\IYE8288\Downloads\logstash-6.3.1\logstash-6.3.1\bin>logstash -f LogstashPipeLine.conf --config.test_and_exit
Sending Logstash's logs to C:/Users/IYE8288/Downloads/logstash-6.3.1/logstash-6.3.1/logs which is now configured via log4j2.properties
[2018-07-12T10:23:05,730][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-07-12T10:23:05,906][FATAL][logstash.runner ] The given configuration is invalid. Reason: Expected one of #, input, filter, output at line 1, column 1 (byte 1) after
[2018-07-12T10:23:05,915][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

My logstash service runs fine .But logs do not display in the output location.

Can you please help me out on this..

12

@anon98066099, please start your own thread for your unrelated problem.

But before you do, google the error message ("The given configuration..."). This exact error has come up more than once before.