Rabbitmq connection error

Elastic Stack Logstash
1

I'm trying to connect logstash to my AMP for Endpoints streaming event server hosted by Cisco.
I had to do a little juggling to get the proper user/pw and the correct URL. But I have them now. I have tested them using a python pika connection and I am able to retrieve events w/o issue so I know they are correct.

I want to configure logstash to pull the event data for me and drop it into elasticsearch. However I am getting an odd error.

My config file:
</>input {
rabbitmq {
host => "export-streaming.amp.cisco.com"
port => 443
user => ""
password => ""
queue => ""
ack => true
durable => true
passive => true
subscription_retry_interval_seconds => 10
}
}
output {

stdout { codec => rubydebug }
}
</>

Now the thing that is bothering me here is the log data I'm getting back and the error location. NOTE: I am connecting to a remote server using port 443.

</>[2018-06-07T15:59:38,038][ERROR][logstash.inputs.rabbitmq ] RabbitMQ connection error, will retry. {:error_message=>"Connection to localhost:5671 refused", :exception=>"MarchHare::ConnectionRefused"}</>

I'm lost as to why I am getting an error for connecting to the local RMQ server, when I am trying to connect to a remote RMQ server. Perhaps I have something incorrectly configured in another config file?

2

Perhaps I have something incorrectly configured in another config file?

Indeed. Do you have any other configuration files in /etc/logstash/conf.d or wherever you store your config files?

3

No. I thought that might be the issue as I had a second config file in that directory. So I emptied the directory of all other config files.
I still get the same error message.
That got me wondering if something else was going on. So I enabled debug with the following

./logstash --log.level debug --path.settings /etc/logstash --path.data /usr/share/logstash/data/

That resulted in the following error appearing in /var/log/logstash/logstash-plain.log

[2018-06-08T09:53:53,193][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ParNew"}
[2018-06-08T09:53:53,193][DEBUG][logstash.instrument.periodicpoller.jvm] collector name {:name=>"ConcurrentMarkSweep"}
[2018-06-08T09:53:53,194][DEBUG][logstash.instrument.periodicpoller.cgroup] Error, cannot retrieve cgroups information {:exception=>"NoMethodError", :message=>"undefined method `[]' for nil:NilClass"}
[2018-06-08T09:53:53,418][DEBUG][logstash.pipeline ] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#<Thread:0x30c60d4@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 sleep>"}

Which has me wondering if this is related to ubuntu 18.04 which is what I am running on at this time.

4

I am still seeing this error. I am uncertain as to what is causing it. I have made some minor changes to the config file in an attempt to resolve it, but without success.

   input {
        rabbitmq{
        host => ["IP_ADDRESS"]
        queue => "event_stream"
        user => ["UN"]
        password => "PW"
        port => 443
        heartbeat => 30
        passive => true
        durable => true
        codec => "json"
        subscription_retry_interval_seconds => 5
        }
}

This error persists:
[ERROR] 2018-06-18 21:37:19.319 [[main]<rabbitmq] rabbitmq - RabbitMQ connection error, will retry. {:error_message=>"Connection to localhost:5672 refused", :exception=>"MarchHare::ConnectionRefused"}

5

I have some additional log data

[2018-06-19T16:59:26,078][ERROR][logstash.inputs.rabbitmq ] RabbitMQ connection error, will retry. {:error_message=>"Connection to localhost:5671 refused", :exception=>"MarchHare::ConnectionRefused", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/march_hare-3.1.1-java/lib/march_hare/session.rb:505:in `converting_rjc_exceptions_to_ruby'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/march_hare-3.1.1-java/lib/march_hare/session.rb:541:in `new_connection_impl'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/march_hare-3.1.1-java/lib/march_hare/session.rb:534:in `build_new_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/march_hare-3.1.1-java/lib/march_hare/session.rb:157:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/march_hare-3.1.1-java/lib/march_hare/session.rb:127:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/march_hare-3.1.1-java/lib/march_hare.rb:24:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-mixin-rabbitmq_connection-5.0.2-java/lib/logstash/plugin_mixins/rabbitmq_connection.rb:201:in `connect'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-mixin-rabbitmq_connection-5.0.2-java/lib/logstash/plugin_mixins/rabbitmq_connection.rb:138:in `connect!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-rabbitmq-6.0.3/lib/logstash/inputs/rabbitmq.rb:183:in `setup!'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-rabbitmq-6.0.3/lib/logstash/inputs/rabbitmq.rb:177:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:512:in `inputworker'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:505:in `block in start_input'"]}
[2018-06-19T16:59:26,955][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"}
[2018-06-19T16:59:26,958][DEBUG][logstash.config.source.local.configpathloader] Skipping the following files while reading config since they don't match the specified glob pattern {:files=>[]}
[2018-06-19T16:59:26,959][DEBUG][logstash.config.source.local.configpathloader] Reading config file {:config_file=>"/etc/logstash/conf.d/test.conf"}
[2018-06-19T16:59:26,960][DEBUG][logstash.agent           ] Converging pipelines state {:actions_count=>0}
[2018-06-19T16:59:27,083][DEBUG][logstash.inputs.rabbitmq ] Connecting to RabbitMQ. Settings: {:vhost=>"/", :hosts=>["export-streaming.com"], :port=>443, :user=>"<UN>", :automatic_recovery=>true, :pass=>"<PW>", :timeout=>0, :heartbeat=>30, :tls=>"TLSv1.2", :tls_certificate_path=>nil, :tls_certificate_password=>nil}
[2018-06-19T16:59:27,085][WARN ][com.rabbitmq.client.TrustEverythingTrustManager] This trust manager trusts every certificate, effectively disabling peer verification. This is convenient for local development but prone to man-in-the-middle attacks. Please see http://www.rabbitmq.com/ssl.html#validating-cerficates to learn more about peer certificate validation.
6

I have confirmed that this is not an issue with Ubuntu 18. I saw this same error on CentOS7 and Ubuntu16:

[2018-06-08T09:53:53,194][DEBUG][logstash.instrument.periodicpoller.cgroup] Error, cannot retrieve cgroups information {:exception=>"NoMethodError", :message=>"undefined method `[]' for nil:NilClass"}
[2018-06-08T09:53:53,418][DEBUG][logstash.pipeline ] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#<Thread:0x30c60d4@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:247 sleep>"}

I also saw this error on all three OS's:
[2018-06-07T15:59:38,038][ERROR][logstash.inputs.rabbitmq ] RabbitMQ connection error, will retry. {:error_message=>"Connection to localhost:5671 refused", :exception=>"MarchHare::ConnectionRefused"}

So I opened an issue here: https://github.com/logstash-plugins/logstash-mixin-rabbitmq_connection/issues/52

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.