Simple question as per the subject line. I need to push data from my source into Hadoop. Can this be done? If yes which supported output plugin could be used?
There's no plugin for this AFAIK.
It seems like it should be possible to use the webhdfs output plugin to do this.
Oh, nice 
Thanks, I will try this out. Anybody did it already? Have any good/bad news about its usage?
I'm using logstash1.5.4, but I can't install logstash-output-webhdfs with 'bin/plugin install logstash-output-webhdfs' or 'bin/plugin install logstash-output-webhdfs-2.0.2.gem', do you have any good way to solve this problem?
Please start your own thread
I was able to use it and test one log...this looks fine!
1 Like
It is not working as expected if I drop the message altogether, since I need to index only few fields from the entire log. If I do remove_field => "message", then the message logged to Hadoop is " %{message}" string. That is literally %{message} string. This looks like a bug!