What is the best solution, using Kafka and Elasticsearch?


I would like to use Elasticsearch for collecting the logs of the our services.
In addtion, I will use Kafka as a messaging queue.

My first question is 'How can I insert the messages in Kafka into Elasticsearch?'
(1) I would like to know the popular options.
(I know Elasticsearch(ver 2.x) does not support River plugins.)

I am using Logstash temporarily.

And also I would like to know.

Kafka and Elasticsearch are currently forming the each Cluster in my environment.
So, I can easily scale them out whenever.

But I installed Logstash on the instance, which Elasticsearch is also installed on.

Do you have any better ideas?
(2) Should I create the instance for ONLY logstash??

Actually I am not infrastructure engineer,
I am game engineer, so I'm not really sure what is the best.

I would be glad if someone give me advices..

I'm not English speaker, so if you don't understand, please tell me what you don't understand.


Elasticsearch 2.2
Logstash 2.2.2

Hope my presentation helps.

My slides are on Slideshare at

We use Kafka extensively, over a trillion messages day.

All of our services use a common logging jar file, which send log data to Kafka.