Logstash A plugin had an unrecoverable error (AWS ARM64/Ubuntu 20.04/Java OpenJDK 11)

Elastic Stack Logstash
1

I have installed ELK using docker from following link https://github.com/deviantony/docker-elk/tree/release-7.x

It works very well before, but recenty I got an issue on several new servers. The issue related with Logstash I installed on the some new servers it no longer send logs to the ELK server (2 earlier servers work very well). All the new servers are running on the AWS ARM64 processor with Ubuntu 20.04 OS (Logstash installed directly on the OS).

Here is my Logstash conf:

ubuntu@ip-172-31-17-36:~$ cat /etc/logstash/conf.d/logstash.conf                
input {                                                                         
  stdin {}                                                                      
  file {                                                                        
    path => "/home/ubuntu/logs/*.log"                                           
    codec => "line"                                                             
    type => "logback"                                                           
  }                                                                             
}                                                                               
output {                                                                        
  elasticsearch {                                                               
    hosts => ["http://my.domain.here:9200"]                                
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"      
    user => "elastic"                                                           
    password => "mypasswordhere"                                           
  }                                                                             
                                                                                
  stdout {                                                                      
    codec => rubydebug                                                          
  }                                                                             
}

And here is the error shown in Kibana Logs:

[logstash.log][ERROR] A plugin had an unrecoverable error. Will restart this plugin.
  Pipeline_id:main
  Plugin: <LogStash::Inputs::File path=>["/home/ubuntu/logs/*.log"], codec=><LogStash::Codecs::Line id=>"line_be8567e0-c2ed-4fb5-bb1e-d5547b6e82ca", enable_metric=>true, charset=>"UTF-8", delimiter=>"\n">, id=>"c268f57e10bc6f90059c4e8a0c16df4e1833157f2a8fc58a01a046529bd84b48", type=>"logback", enable_metric=>true, stat_interval=>1.0, discover_interval=>15, sincedb_write_interval=>15.0, start_position=>"end", delimiter=>"\n", close_older=>3600.0, mode=>"tail", file_completed_action=>"delete", sincedb_clean_after=>1209600.0, file_chunk_size=>32768, file_chunk_count=>140737488355327, file_sort_by=>"last_modified", file_sort_direction=>"asc", exit_after_read=>false, check_archive_validity=>false>
  Error: Operation not permitted - No message available
  Exception: Errno::EPERM
  Stack: org/jruby/RubyFile.java:675:in `chown'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.2/lib/filewatch/helper.rb:41:in `write_atomically'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.2/lib/filewatch/sincedb_collection.rb:232:in `atomic_write'
org/jruby/RubyMethod.java:119:in `call'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.2/lib/filewatch/sincedb_collection.rb:216:in `sincedb_write'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.2/lib/filewatch/sincedb_collection.rb:188:in `flush_at_interval'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.2/lib/filewatch/watch.rb:57:in `subscribe'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.2/lib/filewatch/observing_tail.rb:12:in `subscribe'
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-input-file-4.2.2/lib/logstash/inputs/file.rb:364:in `run'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:405:in `inputworker'
/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:396:in `block in start_input'

Snapshot of errors above:

1908×937 182 KB

FYI, The APMs are working very well.

1845×705 67.9 KB

Could somebody help me how to fix the error?

Any help could be extremely appreciated.

My question on the Stackoverflow: https://stackoverflow.com/questions/64785260/logstash-a-plugin-had-an-unrecoverable-error

2

My longer answers are in this thread.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.