Webhdfs Output plugin does not working with use_kerberos_auth


I am working to deploy logstash in k8s container to output message to hadoop.

According to the documentation, webHDFS and Kerberos authentication are supported.

We try a few setups.

  1. webHDFS without kerberos login -> Success
webhdfs {
    host => "<my_hadoop>"
    port => 9780
    path => "/path/test.log"
    user => "kerberos_user"

  1. webHDFS with kerberos login -> Failed
webhdfs {
    host => "<secure_hadoop>"
    port => 9781
    path => "/path/test.log"
    user => "kerberos_user"
    kerberos_keytab => "user.keytab"
    use_kerberos_auth => "true"

We would like to highlight the error, gssapi is not included in logstash by default

Logstash stopped processing because of an error: (LoadError) no such file to load -- gssapi
org.jruby.exceptions.LoadError: (LoadError) no such file to load -- gssapi

There is workaround to install and copy gssapi lib to hdfs plugins

Similarly, there were login issues in 2018 from other webhdfs user and not get resolved.

Do you still support webhdfs plugin and try to fix these issues?
Do you have a complete image / helm chart to get all plugins installed?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.