Logstash S3 output - NotImplementedError fstat error at startup

Hello,

I'm getting this error when trying to start Logstash with S3 output, could you please help me?

NotImplementedError: fstat unimplemented unsupported or native support failed to load
              size at org/jruby/RubyFile.java:1148
              call at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/s3_expect_100_continue.rb:15
              call at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/s3_bucket_name_restrictions.rb:12
              call at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/s3_bucket_dns.rb:31
              call at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/rest/handler.rb:7
              call at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/user_agent.rb:12
              call at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/plugins/endpoint.rb:41
              call at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/param_validator.rb:21
              call at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/plugins/raise_response_errors.rb:14
              call at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19
              call at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/s3_accelerate.rb:33
              call at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/param_converter.rb:20
              call at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/plugins/response_target.rb:21
      send_request at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/request.rb:70
        put_object at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/base.rb:207
        put_object at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/s3/file_uploader.rb:42
         open_file at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/s3/file_uploader.rb:52
        put_object at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/s3/file_uploader.rb:41
            upload at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/s3/file_uploader.rb:34
       upload_file at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/s3/object.rb:251
  upload_test_file at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-s3-4.0.6/lib/logstash/outputs/s3/write_bucket_permission_validator.rb:43
            valid? at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-s3-4.0.6/lib/logstash/outputs/s3/write_bucket_permission_validator.rb:18
          register at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-s3-4.0.6/lib/logstash/outputs/s3.rb:200
          register at /home/elk/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:8
          register at /home/elk/logstash/logstash-core/lib/logstash/output_delegator.rb:41
   register_plugin at /home/elk/logstash/logstash-core/lib/logstash/pipeline.rb:257
  register_plugins at /home/elk/logstash/logstash-core/lib/logstash/pipeline.rb:268
              each at org/jruby/RubyArray.java:1613
  register_plugins at /home/elk/logstash/logstash-core/lib/logstash/pipeline.rb:268
     start_workers at /home/elk/logstash/logstash-core/lib/logstash/pipeline.rb:277
               run at /home/elk/logstash/logstash-core/lib/logstash/pipeline.rb:207
    start_pipeline at /home/elk/logstash/logstash-core/lib/logstash/agent.rb:389
  • Version: Logstash 5.3.1 (logstash-output-s3-4.0.6)

  • Operating System: Red Hat Enterprise Linux Server release 6.8 (Santiago)

  • Java:
    java version "1.8.0_77"
    Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
    Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)

  • Script to start logstash:

    #!/bin/bash
    . $HOME/.bash_profile
    PIDFILE=$HOME/pids/logstash.pid
    LOGSTASH_CFG=$LOGSTASH_CFG_DIR/logstash.conf
    LS_JAVA_OPTS="-Djava.io.tmpdir=$HOME/work/logstash/tmp"
    nohup $LOGSTASH_HOME/bin/logstash -f $LOGSTASH_CFG > $LOGSTASH_LOG_DIR/logstash.log 2>&1 & echo $! > $PIDFILE

  • Config File:
    input {
    beats {
    port => 5044
    }
    }

    filter{
    }

    output{
    s3{
    access_key_id => "KEY_HERE"
    secret_access_key => "SECRET_HERE"
    bucket => "bucket-name"
    region => "eu-central-1"
    time_file => 5
    }
    }

This seems to be an issue with the glibc/jruby/jdk interaction, in short some libraries are not available and the returned error is the fstat errors. Can you try the following workaround https://github.com/elastic/logstash/issues/3127#issuecomment-247376475

Hi, thank you for your answer. I tried:
export LD_LIBRARY_PATH=/lib64:/usr/lib64:$LD_LIBRARY_PATH
but didnt work, same error.
Is it possible in some way to show what library jruby is trying to load?

starting logstash with strace should give you an idea what library and
where they are loaded.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.