Logstash, Kafka & Elasticsearch

Elastic Stack Logstash
1

Is there a guide for configuration and setup for kafka and logstash connection.

3

please dont do those thread upping messages after waiting a single day. This forum does not come with an SLA.

There is an old blog post when the kafka input/outputs were introduced in logstash, that might help https://www.elastic.co/blog/logstash-kafka-intro

Plus a two blog post series


Apart from that, asking in the logstash group might be more efficent.

--Alex

1 Like
4

Thanks for your input @spinscale.

I want to use the logstash kafka output plugin but it seems I can't link a server which holds my kafka.
Like with the elasticsearch output plugin which has the hosts field(uri).

5

I try to input data from filebeat to logstash..
on logstash my outputs are elasticsearch and kafka..

i tried to add field on my data but it is not showing on kafka.,

6

I also can't output my kafka to elasticsearch.

this is my conf file.

input{
 kafka{
  topics_pattern => "sample_logstash*"
 }
}

output{
 elasticsearch {
  hosts => ["192.168.199.213:9200"]
  index => "sample_kafka-%{+YYYY.MM.dd}"
 }
}
7

Why, what happens? Simply posting your config with indication on what is happening is not helpful.

8

There's no output on elasticsearch. Upon checking the logstash, it just works fine, there no error log.

9

Have you added a stdout to see if there anything coming from kafka?

10

Still, I got this output..

13:32:55.482 [[main]-pipeline-manager] INFO  logstash.outputs.elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//192.168.199.213:9200"]}
13:32:55.485 [[main]-pipeline-manager] INFO  logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
13:32:55.501 [[main]-pipeline-manager] INFO  logstash.pipeline - Pipeline main started
13:32:55.649 [Api Webserver] INFO  logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
11

Is there is nothing else then nothing is being retrieved from kafka.

12

But i have data in kafka. hmm :thinking:

13

Still have a problem with regards to this.

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.