Hello everyone!
I'm new to logstash!
I need to send logs from hdfs-namenode to kafka broker with kerberos.
Here is my configs and full log of logstash.
Please tell me what's wrong?
I think logstash is not using jaas.conf because if I comment sasl_kerberos_service_name it will say that this property is needed but this can be read from jaas.conf...
hdfs-nn.conf
++++++++++++++++++++++
input {
file {
type => "hdp-nn-audit"
path => "/u02/var/log/hadoop/hdfs/hdfs-audit.log"
start_position => end
sincedb_path => "/var/log/logstash/since.db"
}
}
filter{
if [type] == "hdp-nn-audit" {
grok {
match => ["message", "ugi=(?([\w\d-]+))@|ugi=(?([\w\d-]+))/[\w\d-.]+@|ugi=(?([\w\d.-_]+))[\s(]+"]
}
}
}
output {
if [type] == "hdp-nn-audit" {
kafka {
codec => plain {
format => "%{message}"
}
bootstrap_servers => "demo4:6667"
topic_id => "hdfs_audit_log"
security_protocol => "SASL_PLAINTEXT"
sasl_kerberos_service_name => "kafka"
jaas_path => "/opt/logstash-5.2.2/config/kafka_client_jaas.conf"
kerberos_config => "/etc/krb5.conf"
#request_required_acks => 0
#request_timeout_ms => 10000
#producer_type => "async"
#message_send_max_retries => 3
#retry_backoff_ms => 100
#queue_buffering_max_ms => 5000
#queue_enqueue_timeout_ms => 5000
#batch_num_messages => 200
#send_buffer_bytes => 102400
client_id => "hdp-nn-audit"
#partition_key_format => "%{user}"
}
# stdout { codec => rubydebug }
}
}
kafka_client_jaas.conf
++++++++++++++++
KafkaClient {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true
renewTicket=true
serviceName="kafka";
};
bin/logstash -f config/hdfs-nn.conf
is sending somwhere logs, but I can't see them using:
bin/kafka-console-consumer.sh --topic hdfs_audit_log --zookeeper demo4.pro.ru:2181 --security-protocol PLAINTEXTSASL