Directions for installing gssapi to use logstash webhdfs plugin with kerberos

[preddy75 (Scott Preddy)]

Hi,

I am running logstash 6.4.2 (no modifications). I am getting gssapi not found in my scenario below. I am research now how to install gssapi. I would appreciate any information at all on doing this. I have found the ruby gem online for gssapi, but I already see one dependency (ffi) that I do not see in logstash, logstash has ffi-java gem. webhdfs plugin is logstash-output-webhdfs (3.0.6). Please share any information.

I am using webhdfs plugin for logstash and need to use kerberos for auth. After I got everything setup in logstash I get this error on startup: My logstash configuration is after.

[ERROR] 2019-06-10 20:07:00.873 [LogStash::Runner] Logstash - java.lang.IllegalStateException: Logstash stopped processing because of an error: (LoadError) no such file to load -- gssapi

I see in the documentation page for the version of logstash I am using that gssapi needs to be installed -> https://www.elastic.co/guide/en/logstash/6.4/plugins-outputs-webhdfs.html#plugins-outputs-webhdfs-kerberos_keytab

However, I see a bunch of threads on the web where people install it all sorts of ways and even after they install it, things do not work because some sorts of init scripts have not been run, etc.

Is there anywhere that has the proper directions to install gssapi for the logstash installation?

I am using a very vanilla logstash configuration for this:

webhdfs {
host => "edited"
port => 50070
path => "/edited/logstash/dt=%{+YYYY-MM-dd}/logstash-%{+HH}.log"
user => "edited"
use_kerberos_auth => true
kerberos_keytab => "/edited/ca/edited.keytab"
}

I am also using a docker container for logstash and cannot deviate from that in this environment if that makes a difference in configuration steps for gssapi.

[preddy75 (Scott Preddy)]

Here is an update -> I have the following modifications (yes its still messy, but you will get the idea) to get me to the point where I get another known error:

Dockerfile:

FROM docker.elastic.co/logstash/logstash:6.4.2

ARG GIT_SHA
ENV GIT_SHA=${GIT_SHA}

USER 0

WORKDIR /usr/share/logstash
ENV PATH=/usr/share/logstash/bin:$PATH

Ensure Logstash has a UTF-8 locale available.

RUN locale-gen en_US.UTF-8

ENV LANG='en_US.UTF-8' LC_ALL='en_US.UTF-8'

ENV LANG=en_US.UTF-8
LANGUAGE=en_US.UTF-8
LC_CTYPE=en_US.UTF-8
LC_ALL=en_US.UTF-8

COPY gssapi-1.3.0.gem gssapi-1.3.0.gem

COPY ffi-1.11.1.gem ffi-1.11.1.gem

RUN cat /etc/resolv.conf &&
yum clean all &&
yum update -y && yum install sudo gcc make -y &&
yum clean all &&
curl https://curl.haxx.se/download/curl-7.61.0.tar.gz -o curl-7.61.0.tar.gz &&
tar xvzf curl-7.61.0.tar.gz && cd curl-7.61.0 &&
export GEM_HOME=/usr/share/logstash/vendor/bundle/jruby/2.3.0 &&
/usr/share/logstash/vendor/jruby/bin/jruby /usr/share/logstash/vendor/jruby/bin/gem install -V /usr/share/logstash/ffi-1.11.1.gem &&
/usr/share/logstash/vendor/jruby/bin/jruby /usr/share/logstash/vendor/jruby/bin/gem install -V /usr/share/logstash/gssapi-1.3.0.gem

RUN cat Gemfile > Gemfile2

RUN printf 'gem "ffi"\ngem "gssapi"' >> Gemfile2

RUN cp Gemfile2 Gemfile

./configure --with-darwinssl && make && \

echo "Running make install! " && make install

RUN echo "logstash ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers

USER logstash

---

And now when I run the webhdfs plugin I get the same error as this post (that has an interesting solution that I am trying now. If this does not work I have to punt, and would advise logstash staff to include this thread in their docs (in some different format), because this is not working with defined gssapi directions.

[preddy75 (Scott Preddy)]

And just to reiterate: current stack trace:

[INFO ] 2019-06-13 16:21:29.978 [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.4.2"}

[INFO ] 2019-06-13 16:21:37.394 [Converge PipelineAction::Create] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>112, "pipeline.batch.size"=>500, "pipeline.batch.delay"=>50}
[ERROR] 2019-06-13 16:21:38.105 [[main]-pipeline-manager] webhdfs - Webhdfs check request failed. (namenode: , Exception: undefined method read_uint32' for #<FFI::MemoryPointer address=0x7fee6404c7a0 size=4> Did you mean? read_uint read_int read_array_of_uint32 read_array_of_int32 read_pointer read_ulong read_string read_ushort read_array_of_uint64 read_array_of_uint16 get_uint32) [ERROR] 2019-06-13 16:21:38.110 [[main]-pipeline-manager] pipeline - Error registering plugin {:pipeline_id=>"main", :plugin=>"#<LogStash::OutputDelegator:0x725df834>", :error=>"undefined methodread_uint32' for #<FFI::MemoryPointer address=0x7fee6404c7a0 size=4>\nDid you mean? read_uint\n read_int\n read_array_of_uint32\n read_array_of_int32\n read_pointer\n read_ulong\n read_string\n read_ushort\n read_array_of_uint64\n read_array_of_uint16\n get_uint32", :thread=>"#<Thread:0x7f10c5d4 run>"}
[ERROR] 2019-06-13 16:21:38.114 [[main]-pipeline-manager] pipeline - Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<WebHDFS::KerberosError: undefined method read_uint32' for #<FFI::MemoryPointer address=0x7fee6404c7a0 size=4> Did you mean? read_uint read_int read_array_of_uint32 read_array_of_int32 read_pointer read_ulong read_string read_ushort read_array_of_uint64 read_array_of_uint16 get_uint32>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:323:inrequest'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:275:in operate_requests'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:138:inlist'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-webhdfs-3.0.6/lib/logstash/outputs/webhdfs_helper.rb:49:in test_client'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-webhdfs-3.0.6/lib/logstash/outputs/webhdfs.rb:155:inregister'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:102:in register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:46:inregister'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:242:in register_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:253:inblock in register_plugins'", "org/jruby/RubyArray.java:1734:in each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:253:inregister_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:594:in maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:263:instart_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:200:in run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:160:inblock in start'"], :thread=>"#<Thread:0x7f10c5d4 run>"}
[ERROR] 2019-06-13 16:21:38.127 [Converge PipelineAction::Create] agent - Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}
[INFO ] 2019-06-13 16:21:38.402 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600}

and the current suggested fix -> https://github.com/gpaggi/gssapi/commit/69b14cce1966fdae713d274ca19339237eca72b9

which is what I am looking into now

[preddy75 (Scott Preddy)]

And inside currently built image:

bash-4.2$ grep -i -r "read_uint32" .
./vendor/bundle/jruby/2.3.0/gems/gssapi-1.3.0/lib/gssapi/simple.rb: if (ret_flags.read_uint32 & LibGSSAPI::GSS_C_DELEG_FLAG) != 0
./vendor/bundle/jruby/2.3.0/gems/gssapi-1.3.0/lib/gssapi/exceptions.rb: min_stat = min_stat.read_uint32
./vendor/bundle/jruby/2.3.0/gems/gssapi-1.3.0/lib/gssapi/exceptions.rb: @s += "(with major status #{maj}, minor status #{min.read_uint32}\n"
bash-4.2$ pwd
/usr/share/logstash

So going to put this into docker image build (in some form and will retry):

grep -rl read_uint32 /usr/share/logstash | xargs sed -i 's/read_uint32/read_uint/g'

[preddy75 (Scott Preddy)]

So that fix above got past that problem and hit a new one ->

[INFO ] 2019-06-13 18:31:37.995 [Converge PipelineAction::Create] pipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>112, "pipeline.batch.size"=>500, "pipeline.batch.delay"=>50}
[ERROR] 2019-06-13 18:31:38.581 [[main]-pipeline-manager] webhdfs - Webhdfs check request failed. (namenode: , Exception: gss_init_sec_context did not return GSS_S_COMPLETE: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:1000)
)
[ERROR] 2019-06-13 18:31:38.585 [[main]-pipeline-manager] pipeline - Error registering plugin {:pipeline_id=>"main", :plugin=>"#LogStash::OutputDelegator:0x10f3889c", :error=>"gss_init_sec_context did not return GSS_S_COMPLETE: Unspecified GSS failure. Minor code may provide more information\nNo Kerberos credentials available (default cache: KEYRING:persistent:1000)\n", :thread=>"#<Thread:0x1cf29a7d run>"}
[ERROR] 2019-06-13 18:31:38.588 [[main]-pipeline-manager] pipeline - Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<WebHDFS::KerberosError: gss_init_sec_context did not return GSS_S_COMPLETE: Unspecified GSS failure. Minor code may provide more information
No Kerberos credentials available (default cache: KEYRING:persistent:1000)

, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:323:in `request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:275:in `operate_requests'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:138:in `list'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-webhdfs-3.0.6/lib/logstash/outputs/webhdfs_helper.rb:49:in `test_client'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-webhdfs-3.0.6/lib/logstash/outputs/webhdfs.rb:155:in `register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:102:in `register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:46:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:242:in `register_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:253:in `block in register_plugins'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:253:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:594:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:263:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:200:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:160:in `block in start'"], :thread=>"#<Thread:0x1cf29a7d run>"}
[ERROR] 2019-06-13 18:31:38.595 [Converge PipelineAction::Create] agent - Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}