# What is the best solution, using Kafka and Elasticsearch?

**URL:** <https://discuss.elastic.co/t/what-is-the-best-solution-using-kafka-and-elasticsearch/43281>\
**Category:** Elasticsearch\
**Created:** [March 2, 2016, 5:18pm UTC](https://discuss.elastic.co/t/what-is-the-best-solution-using-kafka-and-elasticsearch/43281 "2016-03-02T17:18:36Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![tagia0212](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tagia0212/32/8224_2.png) [@tagia0212](https://discuss.elastic.co/u/tagia0212)\
**Post date:** [March 2, 2016, 5:18pm UTC](https://discuss.elastic.co/t/what-is-the-best-solution-using-kafka-and-elasticsearch/43281/1 "2016-03-02T17:18:36Z")

</div>

Hello.

I would like to use Elasticsearch for collecting the logs of the our services.  
In addtion, I will use Kafka as a messaging queue.

My first question is 'How can I insert the messages in Kafka into Elasticsearch?'  
**(1) I would like to know the popular options.**  
(I know Elasticsearch(ver 2.x) does not support River plugins.)

I am using Logstash temporarily.

And also I would like to know.

Kafka and Elasticsearch are currently forming the each Cluster in my environment.  
So, I can easily scale them out whenever.

But I installed Logstash on the instance, which Elasticsearch is also installed on.

Do you have any better ideas?  
**(2) Should I create the instance for ONLY logstash??**

Actually I am not infrastructure engineer,  
I am game engineer, so I'm not really sure what is the best.

I would be glad if someone give me advices..

I'm not English speaker, so if you don't understand, please tell me what you don't understand.

* * *

[Environment]

Elasticsearch 2.2  
Logstash 2.2.2

---

<div class="post-metadata">

**Author:** ![tinle](https://avatars.discourse-cdn.com/v4/letter/t/c77e96/32.png) [@tinle](https://discuss.elastic.co/u/tinle)\
**Post date:** [March 2, 2016, 5:58pm UTC](https://discuss.elastic.co/t/what-is-the-best-solution-using-kafka-and-elasticsearch/43281/2 "2016-03-02T17:58:08Z")

</div>

Hope my presentation helps.

My slides are on Slideshare at

> **[ELK at LinkedIn - Kafka, scaling, lessons learned](https://www.slideshare.net/TinLe1/elk-atlinked-in)**
>
> How LinkedIn uses and scale ELK clusters using Kafka. Lessons learned. There are useful notes in the PowerPoint version.

We use Kafka extensively, over a trillion messages day.  
[http://www.confluent.io/blog/apache-kafka-hits-1.1-trillion-messages-per-day-joins-the-4-comma-club](http://www.confluent.io/blog/apache-kafka-hits-1.1-trillion-messages-per-day-joins-the-4-comma-club)

All of our services use a common logging jar file, which send log data to Kafka.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:11pm UTC](https://discuss.elastic.co/t/what-is-the-best-solution-using-kafka-and-elasticsearch/43281/3 "2017-07-05T23:11:57Z")

</div>