Can logstash push data into Hadoop?


(Satish) #1

Simple question as per the subject line. I need to push data from my source into Hadoop. Can this be done? If yes which supported output plugin could be used?


(Mark Walkom) #2

There's no plugin for this AFAIK.


(Christian Dahlqvist) #3

It seems like it should be possible to use the webhdfs output plugin to do this.


(Mark Walkom) #4

Oh, nice :slight_smile:


(Satish) #5

Thanks, I will try this out. Anybody did it already? Have any good/bad news about its usage?


(bigfootwalk) #6

I'm using logstash1.5.4, but I can't install logstash-output-webhdfs with 'bin/plugin install logstash-output-webhdfs' or 'bin/plugin install logstash-output-webhdfs-2.0.2.gem', do you have any good way to solve this problem?


(Mark Walkom) #7

Please start your own thread :slight_smile:


(Satish) #8

I was able to use it and test one log...this looks fine!


(Satish) #9

It is not working as expected if I drop the message altogether, since I need to index only few fields from the entire log. If I do remove_field => "message", then the message logged to Hadoop is " %{message}" string. That is literally %{message} string. This looks like a bug!


(system) #10