Logstash S3 output - NotImplementedError fstat error at startup

Elastic Stack Logstash
1

Hello,

I'm getting this error when trying to start Logstash with S3 output, could you please help me?

NotImplementedError: fstat unimplemented unsupported or native support failed to load
              size at org/jruby/RubyFile.java:1148
              call at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/s3_expect_100_continue.rb:15
              call at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/s3_bucket_name_restrictions.rb:12
              call at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/s3_bucket_dns.rb:31
              call at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/rest/handler.rb:7
              call at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/user_agent.rb:12
              call at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/plugins/endpoint.rb:41
              call at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/param_validator.rb:21
              call at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/plugins/raise_response_errors.rb:14
              call at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19
              call at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/s3_accelerate.rb:33
              call at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/aws-sdk-core/plugins/param_converter.rb:20
              call at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/plugins/response_target.rb:21
      send_request at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/request.rb:70
        put_object at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-core-2.3.22/lib/seahorse/client/base.rb:207
        put_object at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/s3/file_uploader.rb:42
         open_file at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/s3/file_uploader.rb:52
        put_object at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/s3/file_uploader.rb:41
            upload at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/s3/file_uploader.rb:34
       upload_file at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/aws-sdk-resources-2.3.22/lib/aws-sdk-resources/services/s3/object.rb:251
  upload_test_file at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-s3-4.0.6/lib/logstash/outputs/s3/write_bucket_permission_validator.rb:43
            valid? at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-s3-4.0.6/lib/logstash/outputs/s3/write_bucket_permission_validator.rb:18
          register at /home/elk/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-s3-4.0.6/lib/logstash/outputs/s3.rb:200
          register at /home/elk/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:8
          register at /home/elk/logstash/logstash-core/lib/logstash/output_delegator.rb:41
   register_plugin at /home/elk/logstash/logstash-core/lib/logstash/pipeline.rb:257
  register_plugins at /home/elk/logstash/logstash-core/lib/logstash/pipeline.rb:268
              each at org/jruby/RubyArray.java:1613
  register_plugins at /home/elk/logstash/logstash-core/lib/logstash/pipeline.rb:268
     start_workers at /home/elk/logstash/logstash-core/lib/logstash/pipeline.rb:277
               run at /home/elk/logstash/logstash-core/lib/logstash/pipeline.rb:207
    start_pipeline at /home/elk/logstash/logstash-core/lib/logstash/agent.rb:389

  • Version: Logstash 5.3.1 (logstash-output-s3-4.0.6)

  • Operating System: Red Hat Enterprise Linux Server release 6.8 (Santiago)

  • Java:
    java version "1.8.0_77"
    Java(TM) SE Runtime Environment (build 1.8.0_77-b03)
    Java HotSpot(TM) 64-Bit Server VM (build 25.77-b03, mixed mode)

  • Script to start logstash:

    #!/bin/bash
    . $HOME/.bash_profile
    PIDFILE=$HOME/pids/logstash.pid
    LOGSTASH_CFG=$LOGSTASH_CFG_DIR/logstash.conf
    LS_JAVA_OPTS="-Djava.io.tmpdir=$HOME/work/logstash/tmp"
    nohup $LOGSTASH_HOME/bin/logstash -f $LOGSTASH_CFG > $LOGSTASH_LOG_DIR/logstash.log 2>&1 & echo $! > $PIDFILE

  • Config File:
    input {
    beats {
    port => 5044
    }
    }

    filter{
    }

    output{
    s3{
    access_key_id => "KEY_HERE"
    secret_access_key => "SECRET_HERE"
    bucket => "bucket-name"
    region => "eu-central-1"
    time_file => 5
    }
    }

2

This seems to be an issue with the glibc/jruby/jdk interaction, in short some libraries are not available and the returned error is the fstat errors. Can you try the following workaround https://github.com/elastic/logstash/issues/3127#issuecomment-247376475

3

Hi, thank you for your answer. I tried:
export LD_LIBRARY_PATH=/lib64:/usr/lib64:$LD_LIBRARY_PATH
but didnt work, same error.
Is it possible in some way to show what library jruby is trying to load?

4

starting logstash with strace should give you an idea what library and
where they are loaded.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.