Avro input to Json output


(no jihun) #1

Hello.

When events created I send it to kafka topic.

now I want to consume that same topic then write it to both of elasticsearch and hdfs directory

for elasticsearsh it simply can be done with logstash ( kafka input, es output ).
This case kafka msg should json formatted.

for hdfs. I have no good idea. and it is my question.

basically I want to use kafka hdfs connector.
but it only supports avro. and logstash not support avro stream.

what about logstash hdfs output ? is that enterprise ready?

events volume is not small , so avro is acceptable and json seems not adoptable. because of file size and network load.

what is your idea for this situation?


(system) #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.