Logstash to Kafka

Elastic Stack Logstash
1

I am trying to set up logstash, client side, so that logstash can consume logs locally on an application server and send it to kafka. I am pretty new to the ELK stack and could use some guidance on how to do this correctly.

The steps I took to install to windows server:

  • installed java
  • Create directory for logstash
  • downloaded and extracted to the logstash directory
  • I created a config file and placed it in the bin directory, of logstash
  • started logstash with this command
    .\logstash.bat -f .\logstash.conf.txt

the config file looks like this:

input {
	file {
		path => "D:\Logs\*.log"
		type => "KawikaTest"
		start_position => "beginning"
		codec => plain{charset => "UTF-16"}        
	}    
}

output {         
	stdout{
		codec => rubydebug
	} 
}

kafka {
	topic_id => "logging_test"
	bootstrap_servers => "SERVERNAME:9092"
}

Here is my main issue. I really am not sure how to configure this. I don't totally understand the options. Any assistance with this would be greatly appreciated.

Also, I'd like to know how I can install this as a service...

Thank You!

2

you should put the kafka in output，just like below:
output {
stdout{
codec => rubydebug
}
kafka {
topic_id => "logging_test"
bootstrap_servers => "SERVERNAME:9092"
}
}

1 Like