SystemCallError, :message=>"Unknown error (SystemCallError)

Elastic Stack Logstash
1

Hello,

We have a server RHEL7 running with logstash, sending the logs to an opensearch with elasticsearch engine, and every few seconds the files are not send with this error:

2023-09-28T06:33:37,702][ERROR][filewatch.tailmode.handlers.grow][pipeline][-] controlled_read general error reading {:path=>"/APP/logs-prod/APPPRD_SRV APP_eu/log.log.log", :exception=>SystemCallError, :message=>"Unknown error (SystemCallError) - /APP/logs-prod/APPPRD_SRV APP_eu/log.log.log", :backtrace=>["org/jruby/RubyIO.java:3153:in `sysread'", " _logstash/logstash-8.9.1/vendor/bundle/jruby/2.6.0/gems/logstash-input-file-4.4.4/lib/filewatch/watched_file.rb:229:in `file_read'", " _logstash/logstash-8.9.1/vendor/bundle/jruby/2.6.0/gems/logstash-input-file-4.4.4/lib/filewatch/watched_file.rb:241:in `read_extract_lines'", " _logstash/logstash-8.9.1/vendor/bundle/jruby/2.6.0/gems/logstash-input-file-4.4.4/lib/filewatch/tail_mode/handlers/base.rb:52:in `block in controlled_read'", "org/jruby/RubyFixnum.java:302:in `times'", " _logstash/logstash-8.9.1/vendor/bundle/jruby/2.6.0/gems/logstash-input-file-4.4.4/lib/filewatch/tail_mode/handlers/base.rb:48:in `controlled_read'", " _logstash/logstash-8.9.1/vendor/bundle/jruby/2.6.0/gems/logstash-input-file-4.4.4/lib/filewatch/tail_mode/handlers/grow.rb:10:in `block in handle_specifically'", "org/jruby/RubyKernel.java:1507:in `loop'", " _logstash/logstash-8.9.1/vendor/bundle/jruby/2.6.0/gems/logstash-input-file-4.4.4/lib/filewatch/tail_mode/handlers/grow.rb:7:in `handle_specifically'", " _logstash/logstash-8.9.1/vendor/bundle/jruby/2.6.0/gems/logstash-input-file-4.4.4/lib/filewatch/tail_mode/handlers/base.rb:25:in `handle'", " _logstash/logstash-8.9.1/vendor/bundle/jruby/2.6.0/gems/logstash-input-file-4.4.4/lib/filewatch/tail_mode/processor.rb:43:in `grow'", " _logstash/logstash-8.9.1/vendor/bundle/jruby/2.6.0/gems/logstash-input-file-4.4.4/lib/filewatch/tail_mode/processor.rb:151:in `block in process_rotation_in_progress'", "org/jruby/RubyArray.java:1865:in `each'", " _logstash/logstash-8.9.1/vendor/bundle/jruby/2.6.0/gems/logstash-input-file-4.4.4/lib/filewatch/tail_mode/processor.rb:142:in `process_rotation_in_progress'", " _logstash/logstash-8.9.1/vendor/bundle/jruby/2.6.0/gems/logstash-input-file-4.4.4/lib/filewatch/tail_mode/processor.rb:71:in `process_all_states'", " _logstash/logstash-8.9.1/vendor/bundle/jruby/2.6.0/gems/logstash-input-file-4.4.4/lib/filewatch/watch.rb:70:in `iterate_on_state'", " _logstash/logstash-8.9.1/vendor/bundle/jruby/2.6.0/gems/logstash-input-file-4.4.4/lib/filewatch/watch.rb:44:in `subscribe'", " _logstash/logstash-8.9.1/vendor/bundle/jruby/2.6.0/gems/logstash-input-file-4.4.4/lib/filewatch/observing_tail.rb:12:in `subscribe'", " _logstash/logstash-8.9.1/vendor/bundle/jruby/2.6.0/gems/logstash-input-file-4.4.4/lib/logstash/inputs/file.rb:370:in `run'", " _logstash/logstash-8.9.1/logstash-core/lib/logstash/java_pipeline.rb:414:in `inputworker'", " _logstash/logstash-8.9.1/logstash-core/lib/logstash/java_pipeline.rb:405:in `block in start_input'"]}

We tried updating the plugins, but without luck.
What else should we try?

Thanks,

2

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

3

We already contacted AWS support and they ask us to confirm that if the error is related to the state file handle errors.

Could you confirm?

Thanks,

4

Can you share your Logstash configuration?

Also, is this path /APP/logs-prod/APPPRD_SRV APP_eu/ a network share?

5

Hey Leandro,

Logstash.conf is:

grep -v '#' logstash.yml
pipeline.ecs_compatibility: disabled
api.enabled: true
api.http.port: 45601

Yes, the path is an EFS network share.

Greetings,

6

This is the logstash.yml file, not the pipeline configuration, you need to share the pipeline configuration with your inputs, filters and outputs.

But this is probably your issue:

Yes, the path is an EFS network share.

The file input does not work well with network shares and you can have some issues from time to time, I don't think there is a fix for it.

Filebeat also does not work well with network shares.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.