WARN ][logstash.inputs.rabbitmq ] Error while setting up connection for rabbitmq input!

Elastic Stack Logstash
1

Hi,
I'm using Logstash6.8.23 and Logstash consumes RabbitMQ message as expected. But Logstash continuously flooding with many warnings as
"[logstash.inputs.rabbitmq ] Error while setting up connection for rabbitmq input! Will retry. {:message=>"#method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - operation not permitted on the default exchange, class-id=50, method-id=20)", :class=>"MarchHare::ChannelAlreadyClosed", :location=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/march_hare-4.3.0-java/lib/march_hare/exceptions.rb:127:in `convert_and_reraise'"}
"
All the exchanges and queue defined in logstash config exists in rabbitmq.
It seems like a new default channel is opened and closed for every event. How to resolve this warning as its flooding the log repo. Please suggest.

Thanks,
Lopa

2

Is there any more of the stack trace in the log?

3

More Details on Logs :

2022-05-12T22:42:07.51+0530 [APP/PROC/WEB/2] OUT [2022-05-12T17:12:07,512][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
2022-05-12T22:42:07.66+0530 [APP/PROC/WEB/1] OUT [2022-05-12T17:12:07,669][DEBUG][logstash.instrument.periodicpoller.cgroup.cpuresource] File /sys/fs/cgroup/cpu/system.slice/garden.service/garden/ed95a46c-0075-470d-40f5-5907/cpu.cfs_period_us cannot be found, try providing an override 'ls.cgroup.cpu.path.override' in the Logstash JAVA_OPTS environment variable
2022-05-12T22:42:07.83+0530 [APP/PROC/WEB/2] OUT [2022-05-12T17:12:07,830][WARN ][logstash.inputs.rabbitmq ] Error while setting up connection for rabbitmq input! Will retry. {:message=>"#method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - operation not permitted on the default exchange, class-id=50, method-id=20)", :class=>"MarchHare::ChannelAlreadyClosed", :location=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/march_hare-4.3.0-java/lib/march_hare/exceptions.rb:127:in convert_and_reraise'"} 2022-05-12T22:42:07.66+0530 [APP/PROC/WEB/1] OUT [2022-05-12T17:12:07,669][DEBUG][logstash.instrument.periodicpoller.cgroup.cpuresource] File /sys/fs/cgroup/cpu/system.slice/garden.service/garden/ed95a46c-0075-470d-40f5-5907/cpu.cfs_quota_us cannot be found, try providing an override 'ls.cgroup.cpu.path.override' in the Logstash JAVA_OPTS environment variable 2022-05-12T22:42:07.66+0530 [APP/PROC/WEB/1] OUT [2022-05-12T17:12:07,669][DEBUG][logstash.instrument.periodicpoller.cgroup.cpuresource] File /sys/fs/cgroup/cpu/system.slice/garden.service/garden/ed95a46c-0075-470d-40f5-5907/cpu.stat cannot be found, try providing an override 'ls.cgroup.cpu.path.override' in the Logstash JAVA_OPTS environment variable 2022-05-12T22:42:07.66+0530 [APP/PROC/WEB/1] OUT [2022-05-12T17:12:07,669][DEBUG][logstash.instrument.periodicpoller.cgroup.cpuacctresource] File /sys/fs/cgroup/cpuacct/system.slice/garden.service/garden/ed95a46c-0075-470d-40f5-5907/cpuacct.usage cannot be found, try providing an override 'ls.cgroup.cpuacct.path.override' in the Logstash JAVA_OPTS environment variable 2022-05-12T22:42:07.71+0530 [APP/PROC/WEB/1] OUT [2022-05-12T17:12:07,718][WARN ][logstash.inputs.rabbitmq ] Error while setting up connection for rabbitmq input! Will retry. {:message=>"#method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - operation not permitted on the default exchange, class-id=50, method-id=20)", :class=>"MarchHare::ChannelAlreadyClosed", :location=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/march_hare-4.3.0-java/lib/march_hare/exceptions.rb:127:in convert_and_reraise'"}
2022-05-12T22:42:08.08+0530 [APP/PROC/WEB/3] OUT [2022-05-12T17:12:08,086][DEBUG][logstash.pipeline ] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#<Thread:0x76a3eb82 sleep>"}
2022-05-12T22:42:08.30+0530 [APP/PROC/WEB/0] OUT [2022-05-12T17:12:08,302][WARN ][logstash.inputs.rabbitmq ] Error while setting up connection for rabbitmq input! Will retry. {:message=>"#method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - operation not permitted on the default exchange, class-id=50, method-id=20)", :class=>"MarchHare::ChannelAlreadyClosed", :location=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/march_hare-4.3.0-java/lib/march_hare/exceptions.rb:127:in convert_and_reraise'"} 2022-05-12T22:42:08.48+0530 [APP/PROC/WEB/3] OUT [2022-05-12T17:12:08,488][WARN ][logstash.inputs.rabbitmq ] Error while setting up connection for rabbitmq input! Will retry. {:message=>"#method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - operation not permitted on the default exchange, class-id=50, method-id=20)", :class=>"MarchHare::ChannelAlreadyClosed", :location=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/march_hare-4.3.0-java/lib/march_hare/exceptions.rb:127:in convert_and_reraise'"}
2022-05-12T22:42:08.71+0530 [APP/PROC/WEB/1] OUT [2022-05-12T17:12:08,717][WARN ][logstash.inputs.rabbitmq ] Error while setting up connection for rabbitmq input! Will retry. {:message=>"#method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - operation not permitted on the default exchange, class-id=50, method-id=20)", :class=>"MarchHare::ChannelAlreadyClosed", :location=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/march_hare-4.3.0-java/lib/march_hare/exceptions.rb:127:in convert_and_reraise'"} 2022-05-12T22:42:08.83+0530 [APP/PROC/WEB/2] OUT [2022-05-12T17:12:08,830][WARN ][logstash.inputs.rabbitmq ] Error while setting up connection for rabbitmq input! Will retry. {:message=>"#method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - operation not permitted on the default exchange, class-id=50, method-id=20)", :class=>"MarchHare::ChannelAlreadyClosed", :location=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/march_hare-4.3.0-java/lib/march_hare/exceptions.rb:127:in convert_and_reraise'"}
2022-05-12T22:42:09.30+0530 [APP/PROC/WEB/0] OUT [2022-05-12T17:12:09,301][WARN ][logstash.inputs.rabbitmq ] Error while setting up connection for rabbitmq input! Will retry. {:message=>"#method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - operation not permitted on the default exchange, class-id=50, method-id=20)", :class=>"MarchHare::ChannelAlreadyClosed", :location=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/march_hare-4.3.0-java/lib/march_hare/exceptions.rb:127:in convert_and_reraise'"} 2022-05-12T22:42:09.48+0530 [APP/PROC/WEB/3] OUT [2022-05-12T17:12:09,487][WARN ][logstash.inputs.rabbitmq ] Error while setting up connection for rabbitmq input! Will retry. {:message=>"#method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - operation not permitted on the default exchange, class-id=50, method-id=20)", :class=>"MarchHare::ChannelAlreadyClosed", :location=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/march_hare-4.3.0-java/lib/march_hare/exceptions.rb:127:in convert_and_reraise'"}
2022-05-12T22:42:09.71+0530 [APP/PROC/WEB/1] OUT [2022-05-12T17:12:09,717][WARN ][logstash.inputs.rabbitmq ] Error while setting up connection for rabbitmq input! Will retry. {:message=>"#method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - operation not permitted on the default exchange, class-id=50, method-id=20)", :class=>"MarchHare::ChannelAlreadyClosed", :location=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/march_hare-4.3.0-java/lib/march_hare/exceptions.rb:127:in convert_and_reraise'"} 2022-05-12T22:42:09.83+0530 [APP/PROC/WEB/2] OUT [2022-05-12T17:12:09,829][WARN ][logstash.inputs.rabbitmq ] Error while setting up connection for rabbitmq input! Will retry. {:message=>"#method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - operation not permitted on the default exchange, class-id=50, method-id=20)", :class=>"MarchHare::ChannelAlreadyClosed", :location=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/march_hare-4.3.0-java/lib/march_hare/exceptions.rb:127:in convert_and_reraise'"}
2022-05-12T22:42:10.30+0530 [APP/PROC/WEB/0] OUT [2022-05-12T17:12:10,301][WARN ][logstash.inputs.rabbitmq ] Error while setting up connection for rabbitmq input! Will retry. {:message=>"#method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - operation not permitted on the default exchange, class-id=50, method-id=20)", :class=>"MarchHare::ChannelAlreadyClosed", :location=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/march_hare-4.3.0-java/lib/march_hare/exceptions.rb:127:in convert_and_reraise'"} 2022-05-12T22:42:10.32+0530 [APP/PROC/WEB/2] OUT [2022-05-12T17:12:10,328][DEBUG][logstash.instrument.periodicpoller.cgroup.cpuresource] File /sys/fs/cgroup/cpu/system.slice/garden.service/garden/aff75ef9-65a4-422a-4b10-e1b5/cpu.cfs_period_us cannot be found, try providing an override 'ls.cgroup.cpu.path.override' in the Logstash JAVA_OPTS environment variable 2022-05-12T22:42:10.32+0530 [APP/PROC/WEB/2] OUT [2022-05-12T17:12:10,328][DEBUG][logstash.instrument.periodicpoller.cgroup.cpuresource] File /sys/fs/cgroup/cpu/system.slice/garden.service/garden/aff75ef9-65a4-422a-4b10-e1b5/cpu.cfs_quota_us cannot be found, try providing an override 'ls.cgroup.cpu.path.override' in the Logstash JAVA_OPTS environment variable 2022-05-12T22:42:10.32+0530 [APP/PROC/WEB/2] OUT [2022-05-12T17:12:10,328][DEBUG][logstash.instrument.periodicpoller.cgroup.cpuresource] File /sys/fs/cgroup/cpu/system.slice/garden.service/garden/aff75ef9-65a4-422a-4b10-e1b5/cpu.stat cannot be found, try providing an override 'ls.cgroup.cpu.path.override' in the Logstash JAVA_OPTS environment variable 2022-05-12T22:42:10.32+0530 [APP/PROC/WEB/2] OUT [2022-05-12T17:12:10,328][DEBUG][logstash.instrument.periodicpoller.cgroup.cpuacctresource] File /sys/fs/cgroup/cpuacct/system.slice/garden.service/garden/aff75ef9-65a4-422a-4b10-e1b5/cpuacct.usage cannot be found, try providing an override 'ls.cgroup.cpuacct.path.override' in the Logstash JAVA_OPTS environment variable 2022-05-12T22:42:10.48+0530 [APP/PROC/WEB/3] OUT [2022-05-12T17:12:10,487][WARN ][logstash.inputs.rabbitmq ] Error while setting up connection for rabbitmq input! Will retry. {:message=>"#method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - operation not permitted on the default exchange, class-id=50, method-id=20)", :class=>"MarchHare::ChannelAlreadyClosed", :location=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/march_hare-4.3.0-java/lib/march_hare/exceptions.rb:127:in convert_and_reraise'"}
2022-05-12T22:42:10.56+0530 [APP/PROC/WEB/0] OUT [2022-05-12T17:12:10,569][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
2022-05-12T22:42:10.56+0530 [APP/PROC/WEB/0] OUT [2022-05-12T17:12:10,569][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
2022-05-12T22:42:10.82+0530 [APP/PROC/WEB/2] OUT [2022-05-12T17:12:10,829][WARN ][logstash.inputs.rabbitmq ] Error while setting up connection for rabbitmq input! Will retry. {:message=>"#method<channel.close>(reply-code=403, reply-text=ACCESS_REFUSED - operation not permitted on the default exchange, class-id=50, method-id=20)", :class=>"MarchHare::ChannelAlreadyClosed", :location=>"/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/march_hare-4.3.0-java/lib/march_hare/exceptions.rb:127:in `convert_and_reraise'"}

Thanks !!

4

Unfortunately, since there is no stack trace, we know that an exception is occurring, but have no idea where in the code it is happening. In the March Hare channel module there are 24 calls to converting_rjc_exceptions_to_ruby (which is where that exception is being logged) and we have no way to find out which one is logging it.

5

Below is the configuration , have added as Logstash input.

rabbitmq
{
host => "{rabbitmq_host}"
port => {rabbitmq_port}
ssl => {use_rmq_ssl}
user => "{rabbitmq_user}"
password => "{rabbitmq_password}"
queue =>"{smoke_queue_name}"
exchange => "{exchange_name}"
exchange_type => "direct"
key => "{routing_key_name}"
durable =>true
metadata_enabled => true
codec => "plain"
vhost => "{rabbitmq_vhost}"
}
}

All are environment variable provided from docker container. Specified exchange and queue are already exist and message are also consumed by Logstash. But warning says , Its trying to connect Default exchange which is causing the warning message . There is no default exchange set here.

6

All I can suggest is to enable log.level DEBUG and see if that provides any more clues as to where the exception is happening.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.