Connection refused to webhdfs


(Sharon Sasporta) #1

Hi,

The Hadoop is up:

vpwrk1@vpelastic:[/home/vpwrk1/var/log/logstash]$ ps -ef | grep hadoop
root 7711 1 0 16:32 ? 00:00:04 /usr/java/jdk1.8.0_74/bin/java -Dproc_datanode -Xmx1000m -Djava.net.preferIPv4Stack=true -Djava.library.path=/usr/local/hadoop/lib/native -Dhadoop.log.dir=/usr/local/hadoop/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/local/hadoop -Dhadoop.id.str=root -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Djava.library.path=/usr/local/hadoop/lib/native -Djava.net.preferIPv4Stack=true -Djava.library.path=/usr/local/hadoop/lib/native -Dhadoop.log.dir=/usr/local/hadoop/logs -Dhadoop.log.file=hadoop-root-datanode-vpelastic.log -Dhadoop.home.dir=/usr/local/hadoop -Dhadoop.id.str=root -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -server -Dhadoop.security.logger=ERROR,RFAS -Dhadoop.security.logger=ERROR,RFAS -Dhadoop.security.logger=ERROR,RFAS -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.datanode.DataNode
root 7886 1 0 16:32 ? 00:00:03 /usr/java/jdk1.8.0_74/bin/java -Dproc_secondarynamenode -Xmx1000m -Djava.net.preferIPv4Stack=true -Djava.library.path=/usr/local/hadoop/lib/native -Dhadoop.log.dir=/usr/local/hadoop/logs -Dhadoop.log.file=hadoop.log -Dhadoop.home.dir=/usr/local/hadoop -Dhadoop.id.str=root -Dhadoop.root.logger=INFO,console -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Stack=true -Djava.library.path=/usr/local/hadoop/lib/native -Djava.net.preferIPv4Stack=true -Djava.library.path=/usr/local/hadoop/lib/native -Dhadoop.log.dir=/usr/local/hadoop/logs -Dhadoop.log.file=hadoop-root-secondarynamenode-vpelastic.log -Dhadoop.home.dir=/usr/local/hadoop -Dhadoop.id.str=root -Dhadoop.root.logger=INFO,RFA -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS -Dhdfs.audit.logger=INFO,NullAppender -Dhadoop.security.logger=INFO,RFAS org.apache.hadoop.hdfs.server.namenode.SecondaryNameNode

Logstash Configuration:

webhdfs {
host => "127.0.0.1" # (required)
port => 50070 # (optional, default: 50070)
path => '/user/logstash/%{type}/logstash%{+yyyy.MM.dd.hh.mm}_%{[@metadata][thread_id]}.json'
user => "vphadoop" # (required)
compression => "snappy"
codec => json
single_file_per_thread => true

When starting the logstash, is failing with following error:

[2018-02-15T16:52:40,710][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>[#<URI::Generic:0x124a864e URL://localhost:9200>]}
[2018-02-15T16:52:40,952][ERROR][logstash.outputs.webhdfs ] Webhdfs check request failed. (namenode: 127.0.0.1:50070, Exception: Failed to connect to host 127.0.0.1:50070, Connection refused - Connection refused)
[2018-02-15T16:52:40,953][ERROR][logstash.pipeline ] Error registering plugin {:plugin=>"#<LogStash::OutputDelegator:0x524e9a9a @namespaced_metric=#<LogStash::Instrument::NamespacedMetric:0x2abc18b2 @metric=#<LogStash::Instrument::Metric:0x5dd43576 @collector=#<LogStash::Instrument::Collector:0x75c27335 @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x53febd5d @store=#<Concurrent::map:0x19cd4618 @default_proc=nil>, @structured_lookup_mutex=#Mutex:0x654ebfd2, @fast_lookup=#<Concurrent::map:0x3f32395 @default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :"36407af94811d744ffe3e94ce9f847a62c4800c9-100"]>, @metric=#<LogStash::Instrument::NamespacedMetric:0x56cf0dd5 @metric=#<LogStash::Instrument::Metric:0x5dd43576 @collector=#<LogStash::Instrument::Collector:0x75c27335 @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x53febd5d @store=#<Concurrent::map:0x19cd4618 @default_proc=nil>, @structured_lookup_mutex=#Mutex:0x654ebfd2, @fast_lookup=#<Concurrent::map:0x3f32395 @default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs]>, @logger=#<LogStash::Logging::Logger:0x5c9ae432 @logger=#Java::OrgApacheLoggingLog4jCore::Logger:0x5378b5ec>, @strategy=#<LogStash::OutputDelegatorStrategies::Legacy:0x5c6dd904 @worker_count=1, @workers=[<LogStash::Outputs::WebHdfs host=>"127.0.0.1", port=>50070, path=>"/user/logstash/%{type}/logstash%{+yyyy.MM.dd.hh.mm}_%{[@metadata][thread_id]}.json", user=>"vphadoop", compression=>"snappy", codec=><LogStash::Codecs::JSON id=>"json_5d658424-a5a8-4712-bcbe-65737d3f419b", enable_metric=>true, charset=>"UTF-8">, single_file_per_thread=>true, id=>"36407af94811d744ffe3e94ce9f847a62c4800c9-100", enable_metric=>true, workers=>1, standby_host=>false, standby_port=>50070, idle_flush_time=>1, flush_size=>500, open_timeout=>30, read_timeout=>30, use_httpfs=>false, retry_known_errors=>true, retry_interval=>0.5, retry_times=>5, snappy_bufsize=>32768, snappy_format=>"stream">], @worker_queue=#SizedQueue:0x6d4fb9e1>, @id="36407af94811d744ffe3e94ce9f847a62c4800c9-100", @metric_events=#<LogStash::Instrument::NamespacedMetric:0x9d04c63 @metric=#<LogStash::Instrument::Metric:0x5dd43576 @collector=#<LogStash::Instrument::Collector:0x75c27335 @agent=nil, @metric_store=#<LogStash::Instrument::MetricStore:0x53febd5d @store=#<Concurrent::map:0x19cd4618 @default_proc=nil>, @structured_lookup_mutex=#Mutex:0x654ebfd2, @fast_lookup=#<Concurrent::map:0x3f32395 @default_proc=nil>>>>, @namespace_name=[:stats, :pipelines, :main, :plugins, :outputs, :"36407af94811d744ffe3e94ce9f847a62c4800c9-100", :events]>, @output_class=LogStash::Outputs::WebHdfs>", :error=>"Failed to connect to host 127.0.0.1:50070, Connection refused - Connection refused"}
[2018-02-15T16:52:41,682][ERROR][logstash.agent ] Pipeline aborted due to error {:exception=>#<WebHDFS::ServerError: Failed to connect to host 127.0.0.1:50070, Connection refused - Connection refused>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/webhdfs-0.8.0/lib/webhdfs/client_v1.rb:351:in `request'",
.........
[2018-02-15T16:52:41,711][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-02-15T16:52:44,699][WARN ][logstash.agent ] stopping pipeline {:id=>"main"}

Any idea? What to check?
It was working till I rebooted the machine.

Thanks
Sharon.


(Magnus B├Ąck) #2

Is the HDFS server really listening on 127.0.0.1:50070? Check with e.g. netstat.


(Sharon Sasporta) #3

Hi Magnus,
Thanks!
Issue was solved.

  1. I had to run the hdfs with the same user as used in logstash output to webhdfs.
  2. Had to run both: start-dfs.sh and start-yarn.sh

Sharon.


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.