# Logstash, Kafka & Elasticsearch

**URL:** <https://discuss.elastic.co/t/logstash-kafka-elasticsearch/99406>\
**Category:** Logstash\
**Created:** [September 5, 2017, 10:49am UTC](https://discuss.elastic.co/t/logstash-kafka-elasticsearch/99406 "2017-09-05T10:49:10Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![jogoinar10](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jogoinar10/32/20068_2.png) [@jogoinar10](https://discuss.elastic.co/u/jogoinar10)\
**Post date:** [September 5, 2017, 10:49am UTC](https://discuss.elastic.co/t/logstash-kafka-elasticsearch/99406/1 "2017-09-05T10:49:10Z")

</div>

Is there a guide for configuration and setup for kafka and logstash connection.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [September 11, 2017, 7:55am UTC](https://discuss.elastic.co/t/logstash-kafka-elasticsearch/99406/3 "2017-09-11T07:55:13Z")

</div>

please dont do those thread upping messages after waiting a single day. This forum does not come with an SLA.

There is an old blog post when the kafka input/outputs were introduced in logstash, that might help [https://www.elastic.co/blog/logstash-kafka-intro](https://www.elastic.co/blog/logstash-kafka-intro)

Plus a two blog post series

> **[Just Enough Kafka for the Elastic Stack, Part 1](https://www.elastic.co/blog/just-enough-kafka-for-the-elastic-stack-part1)**
>
> This post talks about design considerations for integrating Kafka with the Elastic Stack.

  

> **[Just Enough Kafka For The Elastic Stack, Part 2](https://www.elastic.co/blog/just-enough-kafka-for-the-elastic-stack-part2)**
>
> Are you running Apache Kafka with your Elastic Stack? Here's part 2 of the series which talks about operations and production deployment tips.

Apart from that, asking in the logstash group might be more efficent.

--Alex

---

<div class="post-metadata">

**Author:** ![jogoinar10](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jogoinar10/32/20068_2.png) [@jogoinar10](https://discuss.elastic.co/u/jogoinar10)\
**Post date:** [September 13, 2017, 5:37am UTC](https://discuss.elastic.co/t/logstash-kafka-elasticsearch/99406/4 "2017-09-13T05:37:45Z")

</div>

Thanks for your input @spinscale.

I want to use the logstash kafka output plugin but it seems I can't link a server which holds my kafka.  
Like with the elasticsearch output plugin which has the hosts field(uri).

---

<div class="post-metadata">

**Author:** ![jogoinar10](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jogoinar10/32/20068_2.png) [@jogoinar10](https://discuss.elastic.co/u/jogoinar10)\
**Post date:** [September 13, 2017, 10:33am UTC](https://discuss.elastic.co/t/logstash-kafka-elasticsearch/99406/5 "2017-09-13T10:33:52Z")

</div>

I try to input data from filebeat to logstash..  
on logstash my outputs are elasticsearch and kafka..

i tried to add field on my data but it is not showing on kafka.,

---

<div class="post-metadata">

**Author:** ![jogoinar10](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jogoinar10/32/20068_2.png) [@jogoinar10](https://discuss.elastic.co/u/jogoinar10)\
**Post date:** [September 13, 2017, 11:00am UTC](https://discuss.elastic.co/t/logstash-kafka-elasticsearch/99406/6 "2017-09-13T11:00:42Z")

</div>

I also can't output my kafka to elasticsearch.

this is my conf file.

```
input{
 kafka{
  topics_pattern => "sample_logstash*"
 }
}

output{
 elasticsearch {
  hosts => ["192.168.199.213:9200"]
  index => "sample_kafka-%{+YYYY.MM.dd}"
 }
}
```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 13, 2017, 8:28pm UTC](https://discuss.elastic.co/t/logstash-kafka-elasticsearch/99406/7 "2017-09-13T20:28:48Z")

</div>

> [@jogoinar10](#):
>
> I also can't output my kafka to elasticsearch.

Why, what happens? Simply posting your config with indication on _what_ is happening is not helpful.

---

<div class="post-metadata">

**Author:** ![jogoinar10](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jogoinar10/32/20068_2.png) [@jogoinar10](https://discuss.elastic.co/u/jogoinar10)\
**Post date:** [September 14, 2017, 1:10am UTC](https://discuss.elastic.co/t/logstash-kafka-elasticsearch/99406/8 "2017-09-14T01:10:47Z")

</div>

> [@warkolm](#):
>
> Why, what happens? Simply posting your config with indication on what is happening is not helpful.

There's no output on elasticsearch. Upon checking the logstash, it just works fine, there no error log.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 15, 2017, 2:24am UTC](https://discuss.elastic.co/t/logstash-kafka-elasticsearch/99406/9 "2017-09-15T02:24:54Z")

</div>

Have you added a stdout to see if there anything coming from kafka?

---

<div class="post-metadata">

**Author:** ![jogoinar10](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jogoinar10/32/20068_2.png) [@jogoinar10](https://discuss.elastic.co/u/jogoinar10)\
**Post date:** [September 15, 2017, 5:28am UTC](https://discuss.elastic.co/t/logstash-kafka-elasticsearch/99406/10 "2017-09-15T05:28:45Z")

</div>

Still, I got this output..

```
13:32:55.482 [[main]-pipeline-manager] INFO logstash.outputs.elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//192.168.199.213:9200"]}
13:32:55.485 [[main]-pipeline-manager] INFO logstash.pipeline - Starting pipeline {"id"=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>500}
13:32:55.501 [[main]-pipeline-manager] INFO logstash.pipeline - Pipeline main started
13:32:55.649 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=>9600}
```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 15, 2017, 9:13am UTC](https://discuss.elastic.co/t/logstash-kafka-elasticsearch/99406/11 "2017-09-15T09:13:50Z")

</div>

Is there is nothing else then nothing is being retrieved from kafka.

---

<div class="post-metadata">

**Author:** ![jogoinar10](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jogoinar10/32/20068_2.png) [@jogoinar10](https://discuss.elastic.co/u/jogoinar10)\
**Post date:** [September 15, 2017, 10:50am UTC](https://discuss.elastic.co/t/logstash-kafka-elasticsearch/99406/12 "2017-09-15T10:50:42Z")

</div>

But i have data in kafka. hmm 🤔

---

<div class="post-metadata">

**Author:** ![jogoinar10](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jogoinar10/32/20068_2.png) [@jogoinar10](https://discuss.elastic.co/u/jogoinar10)\
**Post date:** [September 19, 2017, 6:42am UTC](https://discuss.elastic.co/t/logstash-kafka-elasticsearch/99406/13 "2017-09-19T06:42:18Z")

</div>

Still have a problem with regards to this.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 17, 2017, 6:42am UTC](https://discuss.elastic.co/t/logstash-kafka-elasticsearch/99406/14 "2017-10-17T06:42:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.