# Send data from Logstash to hadoop

**URL:** <https://discuss.elastic.co/t/send-data-from-logstash-to-hadoop/133976>\
**Category:** Logstash\
**Created:** [May 31, 2018, 6:25am UTC](https://discuss.elastic.co/t/send-data-from-logstash-to-hadoop/133976 "2018-05-31T06:25:36Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![anush\_jayan](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@anush\_jayan](https://discuss.elastic.co/u/anush_jayan)\
**Post date:** [May 31, 2018, 6:25am UTC](https://discuss.elastic.co/t/send-data-from-logstash-to-hadoop/133976/1 "2018-05-31T06:25:36Z")

</div>

hi can help me how can i send data to hadoop or hive using logstash this is my config file

input{  
stdin{}

}

filter{  
grok {  
# patterns\_dir =\> ["/etc/logstash/conf.d/patterns"]  
match =\> { "message" =\> "%{TIMESTAMP\_ISO8601:xxxxxxx},%{IP:client\_xxx.xxx.xxx.xx},%{WORD:xxxxx}&&%{NUMBER:xxxxxxx:int},%{DATESTAMP:xxxxxxx},%{NUMBER:xxxxxxx:float},%{NUMBER:xxxxx:float},%{NUMBER:xxxx:int},%{NUMBER:xxxxxx:int}(:%{NUMBER:xxxxx:int})?,%{WORD:xxxx},%{NUMBER:xxxxxx:float},%{NUMBER:xxxxx:float},%{NUMBER:xxxxxx:float},%{NUMBER:xxxxxx:float},%{NUMBER:xxxx:int},%{NUMBER:xxxxxx:int},%{NUMBER:xxxxxxxx:int},%{NUMBER:xxxxxxx:float},%{NUMBER:xxxxxx:int},%{NUMBER:xxxxxx:int},%{NUMBER:xxxxx:int},%{NUMBER:xxxxxx:int}" }  
}  
}  
output{  
webhdfs{  
host =\> "10.200.234.88"  
port =\> 50070  
user =\> "hduser"  
path =\> "/tmp/logstash.log"  
}  
}

---

<div class="post-metadata">

**Author:** ![anush\_jayan](https://avatars.discourse-cdn.com/v4/letter/a/278dde/32.png) [@anush\_jayan](https://discuss.elastic.co/u/anush_jayan)\
**Post date:** [May 31, 2018, 11:11am UTC](https://discuss.elastic.co/t/send-data-from-logstash-to-hadoop/133976/2 "2018-05-31T11:11:46Z")

</div>

im getting this error in log file of logstash

[2018-05-31T15:32:43,149][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"fb\_apache", :directory=\>"/usr/share/logstash/modules/fb\_apache/configuration"}  
[2018-05-31T15:32:43,156][INFO][logstash.modules.scaffold] Initializing module {:module\_name=\>"netflow", :directory=\>"/usr/share/logstash/modules/netflow/configuration"}  
[2018-05-31T15:32:43,891][ERROR][logstash.outputs.webhdfs] Webhdfs check request failed. (namenode: 10.200.234.88:50070, Exception: Failed to connect to host 10.200.234.88:50070, Connection refused - Connection refused)  
[2018-05-31T15:32:43,900][ERROR][logstash.pipeline] Error registering plugin {:plugin=\>"#\<LogStash::OutputDelegator:0x3489f9a8 @namespaced\_metric=#\<LogStash::Instrument::NamespacedMetric:0x5ff48819 @metric=#\<LogStash::Instrument::Metric:0x3c8a383e @collector=#\<LogStash::Instrument::Collector:0xe5a4479 @agent=nil, @metric\_store=#\<LogStash::Instrument::MetricStore:0x75e5a8a3 @store=#\<Concurrent:🗺0x0000000006566c entries=2 default\_proc=nil\>, @structured\_lookup\_mutex=#Mutex:0xe2a5246, @fast\_lookup=#\<Concurrent:🗺0x00000000065670 entries=57 default\_proc=nil\>\>\>\>, @namespace\_name=[:stats, :pipelines, :main, :plugins, :outputs, :"bf64a8297df974f5b72780197009d24db4775b71-3"]\>, @metric=#\<LogStash::Instrument::NamespacedMetric:0x3fa939f1 @metric=#\<LogStash::Instrument::Metric:0x3c8a383e @collector=#\<LogStash::Instrument::Collector:0xe5a4479 @agent=nil, @metric\_store=#\<LogStash::Instrument::MetricStore:0x75e5a8a3 @store=#\<Concurrent:🗺0x0000000006566c entries=2 default\_proc=nil\>, @structured\_lookup\_mutex=#Mutex:0xe2a5246, @fast\_lookup=#\<Concurrent:🗺0x00000000065670 entries=57 default\_proc=nil\>\>\>\>, @namespace\_name=[:stats, :pipelines, :main, :plugins, :outputs]\>, @logger=#\<LogStash::Logging::Logger:0x3bb5f8d9 @logger=#Java::OrgApacheLoggingLog4jCore::Logger:0x251b1058\>, @out\_counter=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :outputs, :"bf64a8297df974f5b72780197009d24db4775b71-3", :events] key: out value: 0, @in\_counter=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :outputs, :"bf64a8297df974f5b72780197009d24db4775b71-3", :events] key: in value: 0, @strategy=#\<LogStash::OutputDelegatorStrategies::Legacy:0x266076a1 @worker\_count=1, @workers=[\<LogStash::Outputs::WebHdfs host=\>"10.200.234.88", user=\>"hduser", path=\>"/tmp/logstash.log", id=\>"bf64a8297df974f5b72780197009d24db4775b71-3", enable\_metric=\>true, codec=\>\<LogStash::Codecs::Line id=\>"line\_7416d2ae-6cb5-487f-b886-6a6fefce4a75", enable\_metric=\>true, charset=\>"UTF-8", delimiter=\>"\n"\>, workers=\>1, port=\>50070, standby\_host=\>false, standby\_port=\>50070, idle\_flush\_time=\>1, flush\_size=\>500, open\_timeout=\>30, read\_timeout=\>30, use\_httpfs=\>false, single\_file\_per\_thread=\>false, retry\_known\_errors=\>true, retry\_interval=\>0.5, retry\_times=\>5, compression=\>"none", snappy\_bufsize=\>32768, snappy\_format=\>"stream", use\_kerberos\_auth=\>false, use\_ssl\_auth=\>false\>], @worker\_queue=#SizedQueue:0x703f45ee\>, @id="bf64a8297df974f5b72780197009d24db4775b71-3", @time\_metric=LogStash::Instrument::MetricType::Counter - namespaces: [:stats, :pipelines, :main, :plugins, :outputs, :"bf64a8297df974f5b72780197009d24db4775b71-3", :events] key: duration\_in\_millis value: 0, @metric\_events=#\<LogStash::Instrument::NamespacedMetric:0x4263c5ac @metric=#\<LogStash::Instrument::Metric:0x3c8a383e @collector=#\<LogStash::Instrument::Collector:0xe5a4479 @agent=nil, @metric\_store=#\<LogStash::Instrument::MetricStore:0x75e5a8a3 @store=#\<Concurrent:🗺0x0000000006566c entries=2 default\_proc=nil\>, @structured\_lookup\_mutex=#Mutex:0xe2a5246, @fast\_lookup=#\<Concurrent:🗺0x00000000065670 entries=57 default\_proc=nil\>\>\>\>, @namespace\_name=[:stats, :pipelines, :main, :plugins, :outputs, :"bf64a8297df974f5b72780197009d24db4775b71-3", :events]\>, @output\_class=LogStash::Outputs::WebHdfs\>", :error=\>"Failed to connect to host 10.200.234.88:50070, Connection refused - Connection refused"}  
[2018-05-31T15:32:43,904][ERROR][logstash.agent] Pipeline aborted due to error {:exception=\>#\<WebHDFS::ServerError: Failed to connect to host 10.200.234.88:50070, Connection refused - Connection refused\>, :backtrace=\>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/webhdfs-0.8.0/lib/webhdfs/client\_v1.rb:351:in `request'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:275:in`operate\_requests'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/webhdfs-0.8.0/lib/webhdfs/client\_v1.rb:138:in `list'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-webhdfs-3.0.6/lib/logstash/outputs/webhdfs_helper.rb:49:in`test\_client'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-webhdfs-3.0.6/lib/logstash/outputs/webhdfs.rb:155:in `register'", "org/jruby/RubyArray.java:1613:in`each'", "/usr/share/logstash/logstash-core/lib/logstash/output\_delegator\_strategies/legacy.rb:17:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:43:in`register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:290:in `register_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:301:in`register\_plugins'", "org/jruby/RubyArray.java:1613:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:301:in`register\_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:310:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:235:in`run'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:408:in `start\_pipeline'"]}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 28, 2018, 11:12am UTC](https://discuss.elastic.co/t/send-data-from-logstash-to-hadoop/133976/3 "2018-06-28T11:12:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.