(LoadError) no such file to load -- gssapi

[Darshan_Parab]

Hi,

I have configured webhdfs output as show below. I'm getting error that says I don't have gssapi conf webhdfs output.

Pipeline:

input {
beats {
port => 5044
}
}

filter {
}

output {
webhdfs {
host => "hdfsserver"
port => 8020
path => "/user/hadoop/elktest/helloworld.log"
user => "hadoop"
use_kerberos_auth => true
kerberos_keytab => "/opt/hadoop/hadoop.keytab"
}
}

Error:

hdfs_logstash.1.0ip85vorm1yx@SIDCAPASPARK2 | [2019-07-29T07:15:23,530][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, >"pipeline.batch.delay"=>50}
hdfs_logstash.1.ggkrawhiqo23@SIDCAPASPARK2 | start at /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:176
hdfs_logstash.1.0ip85vorm1yx@SIDCAPASPARK2 | warning: thread "[main]-pipeline-manager" terminated with exception (report_on_exception is true):
hdfs_logstash.1.xt1qpca4z7rn@SIDCAPASPARK2 | prepare_client at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-webhdfs 3.0.6/lib/logstash/outputs/webhdfs_helper.rb:27
hdfs_logstash.1.xt1qpca4z7rn@SIDCAPASPARK2 | register at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-webhdfs-3.0.6/lib/logstash/outputs/webhdfs.rb:153
hdfs_logstash.1.0ip85vorm1yx@SIDCAPASPARK2 | LoadError: no such file to load -- gssapi
hdfs_logstash.1.xt1qpca4z7rn@SIDCAPASPARK2 | register at org/logstash/config/ir/compiler/OutputStrategyExt.java:106
hdfs_logstash.1.0ip85vorm1yx@SIDCAPASPARK2 | require at org/jruby/RubyKernel.java:987
hdfs_logstash.1.ggkrawhiqo23@SIDCAPASPARK2 | [2019-07-29T07:14:58,659][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}
hdfs_logstash.1.0ip85vorm1yx@SIDCAPASPARK2 | require at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65
hdfs_logstash.1.ggkrawhiqo23@SIDCAPASPARK2 | [2019-07-29T07:14:58,718][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (LoadError) no such file to load -- gssapi
hdfs_logstash.1.xt1qpca4z7rn@SIDCAPASPARK2 | register at org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:48
hdfs_logstash.1.0ip85vorm1yx@SIDCAPASPARK2 | prepare_client at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-webhdfs-3.0.6/lib/logstash/outputs/webhdfs_helper.rb:27
hdfs_logstash.1.xt1qpca4z7rn@SIDCAPASPARK2 | register_plugin at /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:259
hdfs_logstash.1.0ip85vorm1yx@SIDCAPASPARK2 | register at /usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-webhdfs-3.0.6/lib/logstash/outputs/webhdfs.rb:153
hdfs_logstash.1.xt1qpca4z7rn@SIDCAPASPARK2 | register_plugins at /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:270
hdfs_logstash.1.0ip85vorm1yx@SIDCAPASPARK2 | register at org/logstash/config/ir/compiler/OutputStrategyExt.java:106
hdfs_logstash.1.xt1qpca4z7rn@SIDCAPASPARK2 | each at org/jruby/RubyArray.java:1792
hdfs_logstash.1.0ip85vorm1yx@SIDCAPASPARK2 | register at org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:48
hdfs_logstash.1.xt1qpca4z7rn@SIDCAPASPARK2 | register_plugins at /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:270
hdfs_logstash.1.0ip85vorm1yx@SIDCAPASPARK2 | register_plugin at /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:259
hdfs_logstash.1.xt1qpca4z7rn@SIDCAPASPARK2 | maybe_setup_out_plugins at /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:611
hdfs_logstash.1.0ip85vorm1yx@SIDCAPASPARK2 | register_plugins at /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:270
hdfs_logstash.1.0ip85vorm1yx@SIDCAPASPARK2 | each at org/jruby/RubyArray.java:1792
hdfs_logstash.1.xt1qpca4z7rn@SIDCAPASPARK2 | start_workers at /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:280
hdfs_logstash.1.0ip85vorm1yx@SIDCAPASPARK2 | register_plugins at /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:270
hdfs_logstash.1.xt1qpca4z7rn@SIDCAPASPARK2 | run at /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:217
hdfs_logstash.1.0ip85vorm1yx@SIDCAPASPARK2 | maybe_setup_out_plugins at /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:611
hdfs_logstash.1.xt1qpca4z7rn@SIDCAPASPARK2 | start at /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:176
hdfs_logstash.1.0ip85vorm1yx@SIDCAPASPARK2 | start_workers at /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:280
hdfs_logstash.1.0ip85vorm1yx@SIDCAPASPARK2 | run at /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:217
hdfs_logstash.1.0ip85vorm1yx@SIDCAPASPARK2 | start at /usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:176
hdfs_logstash.1.xt1qpca4z7rn@SIDCAPASPARK2 | [2019-07-29T07:16:13,587][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}
hdfs_logstash.1.xt1qpca4z7rn@SIDCAPASPARK2 | [2019-07-29T07:16:13,672][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (LoadError) no such file to load -- gssapi
hdfs_logstash.1.0ip85vorm1yx@SIDCAPASPARK2 | [2019-07-29T07:15:23,801][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}
hdfs_logstash.1.0ip85vorm1yx@SIDCAPASPARK2 | [2019-07-29T07:15:23,908][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (LoadError) no such file to load -- gssapi

[Darshan_Parab]

I was able to fix the issue using below mentioned discussion thread.

Directions for installing gssapi to use logstash webhdfs plugin with kerberos

Here is an update -> I have the following modifications (yes its still messy, but you will get the idea) to get me to the point where I get another known error: Dockerfile: FROM docker.elastic.co/logstash/logstash:6.4.2 ARG GIT_SHA ENV GIT_SHA=${GIT_SHA} USER 0 WORKDIR /usr/share/logstash ENV PATH=/usr/share/logstash/bin:$PATH Ensure Logstash has a UTF-8 locale available. RUN locale-gen en_US.UTF-8 ENV LANG='en_US.UTF-8' LC_ALL='en_US.UTF-8' ENV LANG=en_US.UTF-8 LANGUAGE=en_US.UTF-8 …

Logstash webhdfs example with Kerberos enabled hadoop cluster

configured the webhdfs_helper.rb that client.kerberos = true /client.kerberos_keytab="xxx" and installed the gssapi but still have such problem: Webhdfs check request failed. (namenode: 10.0.0.37:14000, Exception: undefined method read_uint32' for #<FFI::MemoryPointer address=0xd69f20 size=4>) {:level=>:error} The error reported is: undefined methodread_uint32' for #<FFI::MemoryPointer address=0xd69f20 size=4>

However, now I'm getting below mentioned error.

hdfs_logstash.1.i8axxr7tdhsq@SIDCAPASPARK2    | No Kerberos credentials available: Invalid UID in persistent keyring name
hdfs_logstash.1.i8axxr7tdhsq@SIDCAPASPARK2    | )
hdfs_logstash.1.i8axxr7tdhsq@SIDCAPASPARK2    | [2019-07-29T12:20:24,356][ERROR][logstash.javapipeline    ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<WebHDFS::KerberosError: gss_init_sec_context did not return GSS_S_COMPLETE: Unspecified GSS failure.  Minor code may provide more information
hdfs_logstash.1.i8axxr7tdhsq@SIDCAPASPARK2    | No Kerberos credentials available: Invalid UID in persistent keyring name
hdfs_logstash.1.i8axxr7tdhsq@SIDCAPASPARK2    | >, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:323:in `request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:275:in `operate_requests'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:138:in `list'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-webhdfs-3.0.6/lib/logstash/outputs/webhdfs_helper.rb:49:in `test_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-webhdfs-3.0.6/lib/logstash/outputs/webhdfs.rb:155:in `register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:106:in `register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:48:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:192:in `block in register_plugins'", "org/jruby/RubyArray.java:1792:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:191:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:462:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:204:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:146:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:105:in `block in start'"], :thread=>"#<Thread:0x49a8e1b3 run>"}
hdfs_logstash.1.i8axxr7tdhsq@SIDCAPASPARK2    | [2019-07-29T12:20:24,374][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
hdfs_logstash.1.i8axxr7tdhsq@SIDCAPASPARK2    | [2019-07-29T12:20:24,519][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
hdfs_logstash.1.i8axxr7tdhsq@SIDCAPASPARK2    | [2019-07-29T12:20:29,606][INFO ][logstash.runner          ] Logstash shut down.```