Logstash restarting every 20-30 seconds

Elastic Stack Logstash
1

Log file has the following error
[2018-07-06T07:50:00,389][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

logstash-simple.conf has following code

input { stdin {} }
output {
elasticsearch { hosts => ["ipadd of ES :9200"] }
stdout { codec => rubydebug }
}
Any kind of suggestion is much appreciated.

2

What comes before that in the log?

3

[2018-07-06T07:54:40,440][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
[2018-07-06T07:55:09,375][INFO ][logstash.configmanagement.bootstrapcheck] Using Elasticsearch as config store {:pipeline_id=>["main"], :poll_interval=>"30000000000ns"}
[2018-07-06T07:55:09,445][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: You must set the password using the "xpack.management.elasticsearch.password" in logstash.yml>, :backtrace=>["/usr/share/logstash/x-pack/lib/config_management/elasticsearch_source.rb:40:in initialize'", "/usr/share/logstash/x-pack/lib/config_management/hooks.rb:42:inafter_bootstrap_checks'", "/usr/share/logstash/logstash-core/lib/logstash/event_dispatcher.rb:34:in block in fire'", "/usr/share/logstash/logstash-core/lib/logstash/event_dispatcher.rb:32:infire'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:294:in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:inrun'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:238:in run'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:inrun'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in <main>'"]} [2018-07-06T07:55:09,465][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit [2018-07-06T07:55:36,740][INFO ][logstash.configmanagement.bootstrapcheck] Using Elasticsearch as config store {:pipeline_id=>["main"], :poll_interval=>"30000000000ns"} [2018-07-06T07:55:36,794][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: You must set the password using the "xpack.management.elasticsearch.password" in logstash.yml>, :backtrace=>["/usr/share/logstash/x-pack/lib/config_management/elasticsearch_source.rb:40:ininitialize'", "/usr/share/logstash/x-pack/lib/config_management/hooks.rb:42:in after_bootstrap_checks'", "/usr/share/logstash/logstash-core/lib/logstash/event_dispatcher.rb:34:inblock in fire'", "/usr/share/logstash/logstash-core/lib/logstash/event_dispatcher.rb:32:in fire'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:294:inexecute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:238:inrun'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in run'", "/usr/share/logstash/lib/bootstrap/environment.rb:73:in'"]}
[2018-07-06T07:55:36,811][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

4

ArgumentError: You must set the password using the "xpack.management.elasticsearch.password" in logstash.yml

So... set a password or disable the X-Pack features?

5

After disabled the X-pack, i have the same issue, logstash service is restarting every time,

However below are new logs

[2018-07-06T12:27:13,507][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>RuntimeError, :message=>"Could not fetch the configuration, message: Permission denied - /etc/logstash/conf.d/logstash-simple.conf", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/agent.rb:145:in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:89:inexecute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:363:in block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:inblock in initialize'"]}
[2018-07-06T12:27:13,877][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}

6

Could not fetch the configuration, message: Permission denied - /etc/logstash/conf.d/logstash-simple.conf

This error message is pretty clear, no?

7

Thanks,

I changed the permission in the file, and getting the below logs

[2018-07-06T13:44:40,300][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
[2018-07-06T13:44:40,694][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
Still logstash is reloading.

8

I assume logstash-simple.conf isn't empty? If you increase Logstash's log level it might give additional clues.

9

Yes logstash-simple.conf has the following configuration.
input { stdin {} }
output {
elasticsearch { hosts => ["ES server:9200"] }
stdout { codec => rubydebug }
}

i have a issue of displaying logs now, its been stopped on Friday and i didn't see any other logs after that. Any suggestion is much appriciated.

10

@magnusbaeck , Thank you so much for your time to help me with this issue.
Issue has been resolved after the change in the syslogs & permissions settings.

11

Hi.. I am installing Logstash.. I am getting the below error :

C:\Users\IYE8288\Downloads\logstash-6.3.1\logstash-6.3.1\bin>logstash -f LogstashPipeLine.conf --config.test_and_exit
Sending Logstash's logs to C:/Users/IYE8288/Downloads/logstash-6.3.1/logstash-6.3.1/logs which is now configured via log4j2.properties
[2018-07-12T10:23:05,730][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-07-12T10:23:05,906][FATAL][logstash.runner ] The given configuration is invalid. Reason: Expected one of #, input, filter, output at line 1, column 1 (byte 1) after
[2018-07-12T10:23:05,915][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit

My logstash service runs fine .But logs do not display in the output location.

Can you please help me out on this..

12

@anon98066099, please start your own thread for your unrelated problem.

But before you do, google the error message ("The given configuration..."). This exact error has come up more than once before.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.