# Logstash webhdfs error with kerberos enabled cluster

**URL:** <https://discuss.elastic.co/t/logstash-webhdfs-error-with-kerberos-enabled-cluster/130607>\
**Category:** Logstash\
**Created:** [May 4, 2018, 10:39am UTC](https://discuss.elastic.co/t/logstash-webhdfs-error-with-kerberos-enabled-cluster/130607 "2018-05-04T10:39:43Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Anushke\_Hewawaithara](https://avatars.discourse-cdn.com/v4/letter/a/9de0a6/32.png) [@Anushke\_Hewawaithara](https://discuss.elastic.co/u/Anushke_Hewawaithara)\
**Post date:** [May 4, 2018, 10:39am UTC](https://discuss.elastic.co/t/logstash-webhdfs-error-with-kerberos-enabled-cluster/130607/1 "2018-05-04T10:39:43Z")

</div>

we have logstash pipeline as below to read from kafka topic to webhdfs  
Input = kafka topic  
Output = webhdfs  
I have extracted from source (logstash-6.2.4.tar.gz) and did the testing  
Our hadoop cluster is kerberos enabled cluster

refer attached for more details

---

<div class="post-metadata">

**Author:** ![Anushke\_Hewawaithara](https://avatars.discourse-cdn.com/v4/letter/a/9de0a6/32.png) [@Anushke\_Hewawaithara](https://discuss.elastic.co/u/Anushke_Hewawaithara)\
**Post date:** [May 4, 2018, 10:47am UTC](https://discuss.elastic.co/t/logstash-webhdfs-error-with-kerberos-enabled-cluster/130607/2 "2018-05-04T10:47:46Z")

</div>

# I am getting below error while executing pipeline .

[user@hostname]$ bin/logstash -f /datan1/logstash-6.2.4/bin/pipeline.conf

Sending Logstash's logs to /datan1/logstash-6.2.4/logs which is now configured via log4j2.properties  
[2018-05-04T15:04:06,566][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"netflow", :directory=\>"/datan1/logstash-6.2.4/modules/netflow/configuration"}  
[2018-05-04T15:04:06,607][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"fb\_apache", :directory=\>"/datan1/logstash-6.2.4/modules/fb\_apache/configuration"}  
[2018-05-04T15:04:07,880][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified  
[2018-05-04T15:04:09,193][INFO][logstash.runner] Starting Logstash {"logstash.version"=\>"6.2.4"}  
[2018-05-04T15:04:10,034][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=\>9600}  
[2018-05-04T15:04:15,906][INFO][logstash.pipeline] Starting pipeline {:pipeline\_id=\>"main", "pipeline.workers"=\>32, "pipeline.batch.size"=\>125, "pipeline.batch.delay"=\>50}

---

<div class="post-metadata">

**Author:** ![Anushke\_Hewawaithara](https://avatars.discourse-cdn.com/v4/letter/a/9de0a6/32.png) [@Anushke\_Hewawaithara](https://discuss.elastic.co/u/Anushke_Hewawaithara)\
**Post date:** [May 4, 2018, 10:48am UTC](https://discuss.elastic.co/t/logstash-webhdfs-error-with-kerberos-enabled-cluster/130607/3 "2018-05-04T10:48:00Z")

</div>

**] Error registering plugin {:pipeline\_id=\>"main", :plugin=\>"#\<LogStash::OutputDelegator:0x100380a9 @namespaced\_metric=#\<LogStash::Instrument::NamespacedMetric:0x5ba440f8 @metric=#\<LogStash::Instrument::Metric:0x2affe8ea @collector=#\<LogStash::Instrument::Collector:0x4954cc3d @agent=nil, @metric\_store=#\<LogStash::Instrument::MetricStore:0x45de0b76 @store=#\<Concurrent:🗺0x00000000000fac entries=4 default\_proc=nil\>, @structured\_lookup\_mutex=#Mutex:0x41868944, @fast\_lookup=#\<Concurrent:🗺0x00000000000fb0 entries=63 default\_proc=nil\>\>\>\>, @namespace\_name=[:stats, :pipelines, :main, :plugins, :outputs, :d4fc8a80f489c5060bccfb1317f8c420c21a88b6fd6135075b8f7131c356cf29]\>, @metric=#\<LogStash::Instrument::NamespacedMetric:0x5c5513c0 @metric=#\<LogStash::Instrument::Metric:0x2affe8ea @collector=#\<LogStash::Instrument::Collector:0x4954cc3d @agent=nil, @metric\_store=#\<LogStash::Instrument::MetricStore:0x45de0b76 @store=#\<Concurrent:🗺0x00000000000fac entries=4 default\_proc=nil\>, @structured\_lookup\_mutex=#Mutex:0x41868944, @fast\_lookup=#\<Concurrent:🗺0x00000000000fb0 entries=63 default\_proc=nil\>\>\>\>, @namespace\_name=[:stats, :pipelines, :main, :plugins, :outputs]\>, @out\_counter=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: out value:0, @strategy=#\<LogStash::OutputDelegatorStrategies::Legacy:0x636843ca @worker\_count=1, @workers=[\<LogStash::Outputs::WebHdfs host=\>"xxx.xx.xx.xx", port=\>50070, path=\>"/user/logstash/ocs\_cdr\_data/dt=%{+YYYY-MM-dd}/ocs\_cdr\_data-%{+HH}.log", user=\>"user", use\_kerberos\_auth=\>true, kerberos\_keytab=\>"/home/user/user.keytab", id=\>"d4fc8a80f489c5060bccfb1317f8c420c21a88b6fd6135075b8f7131c356cf29", enable\_metric=\>true, codec=\>\<LogStash::Codecs::Line id=\>"line\_80e1b3a5-9a38-4674-85cc-c7e519b32c2e", enable\_metric=\>true, charset=\>"UTF-8", delimiter=\>"\n"\>, workers=\>1, standby\_host=\>false, standby\_port=\>50070, idle\_flush\_time=\>1, flush\_size=\>500, open\_timeout=\>30, read\_timeout=\>30, use\_httpfs=\>false, single\_file\_per\_thread=\>false, retry\_known\_errors=\>true, retry\_interval=\>0.5, retry\_times=\>5, compression=\>"none", snappy\_bufsize=\>32768, snappy\_format=\>"stream", use\_ssl\_auth=\>false\>], @worker\_queue=#SizedQueue:0x3a5f35e0\>, @in\_counter=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: in value:0, @id="d4fc8a80f489c5060bccfb1317f8c420c21a88b6fd6135075b8f7131c356cf29", @time\_metric=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: duration\_in\_millis value:0, @metric\_events=#\<LogStash::Instrument::NamespacedMetric:0x25b6080a @metric=#\<LogStash::Instrument::Metric:0x2affe8ea @collector=#\<LogStash::Instrument::Collector:0x4954cc3d @agent=nil, @metric\_store=#\<LogStash::Instrument::MetricStore:0x45de0b76 @store=#\<Concurrent:🗺0x00000000000fac entries=4 default\_proc=nil\>, @structured\_lookup\_mutex=#Mutex:0x41868944, @fast\_lookup=#\<Concurrent:🗺0x00000000000fb0 entries=63 default\_proc=nil\>\>\>\>, @namespace\_name=[:stats, :pipelines, :main, :plugins, :outputs, :d4fc8a80f489c5060bccfb1317f8c420c21a88b6fd6135075b8f7131c356cf29, :events]\>, @output\_class=LogStash::Outputs::WebHdfs\>", :error=\>"uninitialized constant GSSAPI::GssApiError::LibGSSAPI\nDid you mean? GSSAPI", :thread=\>"#\<Thread:0x773117db run\>"}**  
**[2018-05-04T15:04:19,411][ERROR][logstash.pipeline] Pipeline aborted due to error {:pipeline\_id=\>"main", :exception=\>#\<NameError: uninitialized constant GSSAPI::GssApiError::LibGSSAPI**  
**Did you mean? GSSAPI\>, :backtrace=\>["org/jruby/RubyModule.java:3343:in `const_missing'", "/datan1/logstash-6.2.4/vendor/bundle/jruby/2.3.0/gems/webhdfs-0.8.0/lib/webhdfs/exceptions.rb:9:in`class:GssApiError'", "/datan1/logstash-6.2.4/vendor/bundle/jruby/2.3.0/gems/webhdfs-0.8.0/lib/webhdfs/exceptions.rb:7:in `<module:GSSAPI>'", "/datan1/logstash-6.2.4/vendor/bundle/jruby/2.3.0/gems/webhdfs-0.8.0/lib/webhdfs/exceptions.rb:6:in`'", "org/jruby/RubyKernel.java:955:in `require'", "uri:classloader:/jruby/kernel/kernel.rb:13:in`require\_relative'", "/datan1/logstash-6.2.4/vendor/bundle/jruby/2.3.0/gems/webhdfs-0.8.0/lib/webhdfs/client\_v1.rb:1:in `<main>'", "org/jruby/RubyKernel.java:955:in`require'", "uri:classloader:/jruby/kernel/kernel.rb:13:in `require_relative'", "/datan1/logstash-6.2.4/vendor/bundle/jruby/2.3.0/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:6:in`'", "org/jruby/RubyKernel.java:955:in `require'", "/datan1/logstash-6.2.4/vendor/bundle/jruby/2.3.0/gems/polyglot-0.3.5/lib/polyglot.rb:65:in`require'", "/datan1/logstash-6.2.4/vendor/bundle/jruby/2.3.0/gems/webhdfs-0.8.0/lib/webhdfs/client.rb:1:in `<main>'", "org/jruby/RubyKernel.java:955:in`require'", "/datan1/logstash-6.2.4/vendor/bundle/jruby/2.3.0/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'", "/datan1/logstash-6.2.4/vendor/bundle/jruby/2.3.0/gems/logstash-output-webhdfs-3.0.6/lib/logstash/outputs/webhdfs_helper.rb:12:in`load\_module'", "/datan1/logstash-6.2.4/vendor/bundle/jruby/2.3.0/gems/logstash-output-webhdfs-3.0.6/lib/logstash/outputs/webhdfs.rb:135:in `register'", "org/jruby/RubyArray.java:1734:in`each'", "/datan1/logstash-6.2.4/logstash-core/lib/logstash/output\_delegator\_strategies/legacy.rb:17:in `register'", "/datan1/logstash-6.2.4/logstash-core/lib/logstash/output_delegator.rb:42:in`register'", "/datan1/logstash-6.2.4/logstash-core/lib/logstash/pipeline.rb:342:in `register_plugin'", "/datan1/logstash-6.2.4/logstash-core/lib/logstash/pipeline.rb:353:in`block in register\_plugins'", "org/jruby/RubyArray.java:1734:in `each'", "/datan1/logstash-6.2.4/logstash-core/lib/logstash/pipeline.rb:353:in`register\_plugins'", "/datan1/logstash-6.2.4/logstash-core/lib/logstash/pipeline.rb:730:in `maybe_setup_out_plugins'", "/datan1/logstash-6.2.4/logstash-core/lib/logstash/pipeline.rb:363:in`start\_workers'", "/datan1/logstash-6.2.4/logstash-core/lib/logstash/pipeline.rb:290:in `run'", "/datan1/logstash-6.2.4/logstash-core/lib/logstash/pipeline.rb:250:in`block in start'"], :thread=\>"#\<Thread:0x773117db run\>"}**  
**[2018-05-04T15:04:19,465][ERROR][logstash.agent] Failed to execute action {:id=\>:main, :action\_type=\>LogStash::ConvergeResult::FailedAction, :message=\>"Could not execute action: LogStash::PipelineAction::Create/pipeline\_id:main, action\_result: false", :backtrace=\>nil}**  
**] Pipeline aborted due to error {:pipeline\_id=\>"main", :exception=\>#\<NameError: uninitialized constant GSSAPI::GssApiError::LibGSSAPI**  
**strong text**  
**==============**

---

<div class="post-metadata">

**Author:** ![Anushke\_Hewawaithara](https://avatars.discourse-cdn.com/v4/letter/a/9de0a6/32.png) [@Anushke\_Hewawaithara](https://discuss.elastic.co/u/Anushke_Hewawaithara)\
**Post date:** [May 4, 2018, 10:48am UTC](https://discuss.elastic.co/t/logstash-webhdfs-error-with-kerberos-enabled-cluster/130607/4 "2018-05-04T10:48:22Z")

</div>

my pipeline config as below  
[username@hostname]$ cat /datan1/logstash-6.2.4/bin/pipeline.conf  
input {  
kafka{  
group\_id =\> "logstash"  
jaas\_path =\> "/usr/logstash/"  
sasl\_kerberos\_service\_name =\> "kafka"  
kerberos\_config =\> "/etc/krb5.conf"  
auto\_offset\_reset =\> "earliest"  
topics =\> ["ocs\_cdr\_data"]  
codec =\> "json"  
bootstrap\_servers =\> "ip1:9092,ip2:9092,ip3:9092,ip4:9092"  
type =\> "ocs\_cdr\_data"  
}}  
output {

webhdfs {  
host =\> "ip.xx.xx.x" # (required)  
port =\> 50070 # (optional, default: 50070)  
path =\> "/user/logstash/ocs\_cdr\_data/dt=%{+YYYY-MM-dd}/ocs\_cdr\_data-%{+HH}.log" # (required)  
user =\> "username" # (required)  
use\_kerberos\_auth =\> "true"  
kerberos\_keytab =\> "/home/username/username.keytab"  
}  
}

---

<div class="post-metadata">

**Author:** ![Anushke\_Hewawaithara](https://avatars.discourse-cdn.com/v4/letter/a/9de0a6/32.png) [@Anushke\_Hewawaithara](https://discuss.elastic.co/u/Anushke_Hewawaithara)\
**Post date:** [May 4, 2018, 10:48am UTC](https://discuss.elastic.co/t/logstash-webhdfs-error-with-kerberos-enabled-cluster/130607/5 "2018-05-04T10:48:40Z")

</div>

=======================================  
jaas.conf as below  
KafkaClient {  
com.sun.security.auth.module.Krb5LoginModule required  
useKeyTab=true  
storeKey=true  
useTicketCache=true  
renewTicket=true  
serviceName="kafka  
keyTab="/home/username/username.keytab"  
principal="username@DOMAIN";  
};  
~

---

<div class="post-metadata">

**Author:** ![Anushke\_Hewawaithara](https://avatars.discourse-cdn.com/v4/letter/a/9de0a6/32.png) [@Anushke\_Hewawaithara](https://discuss.elastic.co/u/Anushke_Hewawaithara)\
**Post date:** [May 4, 2018, 10:48am UTC](https://discuss.elastic.co/t/logstash-webhdfs-error-with-kerberos-enabled-cluster/130607/6 "2018-05-04T10:48:54Z")

</div>

~pipeline.conf validated successfully as below  
[username@hostname] $ bin/logstash -f /datan1/logstash-6.2.4/bin/pipeline.conf --config.test\_and\_exit  
Sending Logstash's logs to /datan1/logstash-6.2.4/logs which is now configured via log4j2.properties  
[2018-05-04T15:59:45,177][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"netflow", :directory=\>"/datan1/logstash-6.2.4/modules/netflow/configuration"}  
[2018-05-04T15:59:45,195][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"fb\_apache", :directory=\>"/datan1/logstash-6.2.4/modules/fb\_apache/configuration"}  
[2018-05-04T15:59:45,709][WARN][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified  
Configuration OK  
[2018-05-04T15:59:48,478][INFO][logstash.runner] Using config.test\_and\_exit mode. Config Validation Result: OK. Exiting Logstash

---

<div class="post-metadata">

**Author:** ![Anushke\_Hewawaithara](https://avatars.discourse-cdn.com/v4/letter/a/9de0a6/32.png) [@Anushke\_Hewawaithara](https://discuss.elastic.co/u/Anushke_Hewawaithara)\
**Post date:** [May 4, 2018, 10:49am UTC](https://discuss.elastic.co/t/logstash-webhdfs-error-with-kerberos-enabled-cluster/130607/7 "2018-05-04T10:49:07Z")

</div>

please assist

---

<div class="post-metadata">

**Author:** ![Anushke\_Hewawaithara](https://avatars.discourse-cdn.com/v4/letter/a/9de0a6/32.png) [@Anushke\_Hewawaithara](https://discuss.elastic.co/u/Anushke_Hewawaithara)\
**Post date:** [May 4, 2018, 10:50am UTC](https://discuss.elastic.co/t/logstash-webhdfs-error-with-kerberos-enabled-cluster/130607/8 "2018-05-04T10:50:29Z")

</div>

error

Pipeline aborted due to error {:pipeline\_id=\>"main", :exception=\>#\<NameError: uninitialized constant GSSAPI::GssApiError::LibGSSAPI  
Did you mean? GSSAPI\>, :backtrace=\>["org/jruby/RubyModule.java:3343:in const\_missing'", "/datan1/logstash-

---

<div class="post-metadata">

**Author:** ![Anushke\_Hewawaithara](https://avatars.discourse-cdn.com/v4/letter/a/9de0a6/32.png) [@Anushke\_Hewawaithara](https://discuss.elastic.co/u/Anushke_Hewawaithara)\
**Post date:** [May 9, 2018, 4:00am UTC](https://discuss.elastic.co/t/logstash-webhdfs-error-with-kerberos-enabled-cluster/130607/9 "2018-05-09T04:00:45Z")

</div>

Can Anyone help on this ?

---

<div class="post-metadata">

**Author:** ![Anushke\_Hewawaithara](https://avatars.discourse-cdn.com/v4/letter/a/9de0a6/32.png) [@Anushke\_Hewawaithara](https://discuss.elastic.co/u/Anushke_Hewawaithara)\
**Post date:** [May 11, 2018, 10:25am UTC](https://discuss.elastic.co/t/logstash-webhdfs-error-with-kerberos-enabled-cluster/130607/10 "2018-05-11T10:25:58Z")

</div>

Can anyone help ?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 8, 2018, 10:26am UTC](https://discuss.elastic.co/t/logstash-webhdfs-error-with-kerberos-enabled-cluster/130607/11 "2018-06-08T10:26:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.