Logstash failing and restarting every 13 seconds

Elastic Stack Logstash
1

The following error is occurring on one node in a three node elasticsearch/logstash cluster. It was functioning normally and apparently something changed recently (unaware of any work on the server at that time and not a system that people normally interact with). Anyway cpu jumped about this time and these errors started appearing, there are no issues with the other two nodes in the cluster and I compared permissions on the logstash files in /etc/ in /usr/share/logstash and in our logs /opt/... The permissions are all the same. additionally if logstash didnt have write access to the location listed, it wouldnt be able to record the below error so something odd going on.

[2018-12-19T17:41:38,931][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ArgumentError: Path "/opt/elkstack/data/logstash/dfq-00" must be a writable directory. It is not writable.>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:448:in validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:230:invalidate_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:141:in block in validate_all'", "org/jruby/RubyHash.java:1343:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:140:in validate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:264:inexecute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in run'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:219:inrun'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in run'", "/usr/share/logstash/lib/bootstrap/environment.rb:67:in'"]}

[2018-12-19T17:41:38,942][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: org.jruby.exceptions.RaiseException: (SystemExit) exit

2

I should also mention that I am running logstash v.6.2.1 in the cluster.

3

Would indicate something is still amiss with permissions.

4

I have set every directory in the path including /opt to 777 and the log file itself and its still throwing the same errors.

5

chmod 777 is a bad idea... You should could correct that, directories should have 755 and files should have 644.

You should start with a chown -R logstash: /opt/elkstack that would correct your permission problem.

6

I absolutely agree that 777 is a very unwise configuration. I did it to prove a point, I also confirmed that everything below /opt is owned by logstash. Again I have two other nodes that have the same configuration but are not reporting the same errors. I am just wondering what else could be wrong that could produce the same error? It isn't a permissions issue with the logging directory.

7

Hmm and you are sure the logstash is running as that user that owns the /opt/elkstack dir? What happens if you run logstash as root (just for test of course)

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.