I have installed ELK using docker from following link https://github.com/deviantony/docker-elk/tree/release-7.x
It works very well before, but recenty I got an issue on several new servers. The issue related with Logstash I installed on the some new servers it no longer send logs to the ELK server (2 earlier servers work very well). All the new servers are running on the AWS ARM64 processor with Ubuntu 20.04 OS (Logstash installed directly on the OS).
Here is my Logstash conf:
ubuntu@ip-172-31-17-36:~$ cat /etc/logstash/conf.d/logstash.conf
input {
stdin {}
file {
path => "/home/ubuntu/logs/*.log"
codec => "line"
type => "logback"
}
}
output {
elasticsearch {
hosts => ["http://my.domain.here:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
user => "elastic"
password => "mypasswordhere"
}
stdout {
codec => rubydebug
}
}
And here is the error shown in Kibana Logs:
[logstash.log][ERROR] A plugin had an unrecoverable error. Will restart this plugin.
Pipeline_id:main
Plugin: <LogStash::Inputs::File path=>["/home/ubuntu/logs/*.log"], codec=><LogStash::Codecs::Line id=>"line_be8567e0-c2ed-4fb5-bb1e-d5547b6e82ca", enable_metric=>true, charset=>"UTF-8", delimiter=>"\n">, id=>"c268f57e10bc6f90059c4e8a0c16df4e1833157f2a8fc58a01a046529bd84b48", type=>"logback", enable_metric=>true, stat_interval=>1.0, discover_interval=>15, sincedb_write_interval=>15.0, start_position=>"end", delimiter=>"\n", close_older=>3600.0, mode=>"tail", file_completed_action=>"delete", sincedb_clean_after=>1209600.0, file_chunk_size=>32768, file_chunk_count=>140737488355327, file_sort_by=>"last_modified", file_sort_direction=>"asc", exit_after_read=>false, check_archive_validity=>false>
Error: Operation not permitted - No message available
Exception: Errno::EPERM
Stack: org/jruby/RubyFile.java:675:in `chown'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.2/lib/filewatch/helper.rb:41:in `write_atomically'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.2/lib/filewatch/sincedb_collection.rb:232:in `atomic_write'
org/jruby/RubyMethod.java:119:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.2/lib/filewatch/sincedb_collection.rb:216:in `sincedb_write'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.2/lib/filewatch/sincedb_collection.rb:188:in `flush_at_interval'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.2/lib/filewatch/watch.rb:57:in `subscribe'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.2/lib/filewatch/observing_tail.rb:12:in `subscribe'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.2/lib/logstash/inputs/file.rb:364:in `run'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:405:in `inputworker'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:396:in `block in start_input'
Snapshot of errors above:
FYI, The APMs are working very well.
Could somebody help me how to fix the error?
Any help could be extremely appreciated.
My question on the Stackoverflow: https://stackoverflow.com/questions/64785260/logstash-a-plugin-had-an-unrecoverable-error