Logstash, Kafka & Elasticsearch

Is there a guide for configuration and setup for kafka and logstash connection.

please dont do those thread upping messages after waiting a single day. This forum does not come with an SLA.

There is an old blog post when the kafka input/outputs were introduced in logstash, that might help https://www.elastic.co/blog/logstash-kafka-intro

Plus a two blog post series

Apart from that, asking in the logstash group might be more efficent.


1 Like

Thanks for your input @spinscale.

I want to use the logstash kafka output plugin but it seems I can't link a server which holds my kafka.
Like with the elasticsearch output plugin which has the hosts field(uri).

I try to input data from filebeat to logstash..
on logstash my outputs are elasticsearch and kafka..

i tried to add field on my data but it is not showing on kafka.,

I also can't output my kafka to elasticsearch.

this is my conf file.

  topics_pattern => "sample_logstash*"

 elasticsearch {
  hosts => [""]
  index => "sample_kafka-%{+YYYY.MM.dd}"

Why, what happens? Simply posting your config with indication on what is happening is not helpful.

There's no output on elasticsearch. Upon checking the logstash, it just works fine, there no error log.

Have you added a stdout to see if there anything coming from kafka?

Still, I got this output..

13:32:55.482 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//"]}
13:32:55.485 [[main]-pipeline-manager] INFO  logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
13:32:55.501 [[main]-pipeline-manager] INFO  logstash.pipeline - Pipeline main started
13:32:55.649 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9600}

Is there is nothing else then nothing is being retrieved from kafka.

But i have data in kafka. hmm :thinking:

Still have a problem with regards to this.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.