Webhdfs Output plugin does not working with use_kerberos_auth

Elastic Stack Logstash
1

Hello,

I am working to deploy logstash in k8s container to output message to hadoop.
https://github.com/elastic/helm-charts

According to the documentation, webHDFS and Kerberos authentication are supported.
https://www.elastic.co/guide/en/logstash/current/plugins-outputs-webhdfs.html#plugins-outputs-webhdfs-use_kerberos_auth

We try a few setups.

  1. webHDFS without kerberos login -> Success
webhdfs {
    host => "<my_hadoop>"
    port => 9780
    path => "/path/test.log"
    user => "kerberos_user"
}
  1. webHDFS with kerberos login -> Failed
webhdfs {
    host => "<secure_hadoop>"
    port => 9781
    path => "/path/test.log"
    user => "kerberos_user"
    kerberos_keytab => "user.keytab"
    use_kerberos_auth => "true"
}

We would like to highlight the error, gssapi is not included in logstash by default

Logstash stopped processing because of an error: (LoadError) no such file to load -- gssapi
org.jruby.exceptions.LoadError: (LoadError) no such file to load -- gssapi

There is workaround to install and copy gssapi lib to hdfs plugins
https://discuss.elastic.co/t/loaderror-no-such-file-to-load-gssapi/192619

Similarly, there were login issues in 2018 from other webhdfs user and not get resolved.
https://discuss.elastic.co/t/webhdfs-output-plugin-does-not-login-with-keytab/132164
https://github.com/logstash-plugins/logstash-output-webhdfs/issues/29

Do you still support webhdfs plugin and try to fix these issues?
Do you have a complete image / helm chart to get all plugins installed?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.