What is the best solution, using Kafka and Elasticsearch?


(Shohei Miyashita) #1

Hello.

I would like to use Elasticsearch for collecting the logs of the our services.
In addtion, I will use Kafka as a messaging queue.

My first question is 'How can I insert the messages in Kafka into Elasticsearch?'
(1) I would like to know the popular options.
(I know Elasticsearch(ver 2.x) does not support River plugins.)

I am using Logstash temporarily.

And also I would like to know.

Kafka and Elasticsearch are currently forming the each Cluster in my environment.
So, I can easily scale them out whenever.

But I installed Logstash on the instance, which Elasticsearch is also installed on.

Do you have any better ideas?
(2) Should I create the instance for ONLY logstash??

Actually I am not infrastructure engineer,
I am game engineer, so I'm not really sure what is the best.

I would be glad if someone give me advices..

I'm not English speaker, so if you don't understand, please tell me what you don't understand.


[Environment]

Elasticsearch 2.2
Logstash 2.2.2


(Tin Le) #2

Hope my presentation helps.

My slides are on Slideshare at

We use Kafka extensively, over a trillion messages day.
http://www.confluent.io/blog/apache-kafka-hits-1.1-trillion-messages-per-day-joins-the-4-comma-club

All of our services use a common logging jar file, which send log data to Kafka.


(system) #3