Hi, I'm working on a project to containerise all of our applications to remove the reliance on physical devices (as I'm sure half the world is doing right now!).
We've moved most of our application estate to Docker/K8s and with it the logstash processes.
I am now working on moving our RabbitMQ broker to K8s, and have assigned some non-standard ports to the RMQ cluster. This part is working (ie, I can write messages to an exchange, collect from a queue etc), however I cannot get Logstash to connect to Rabbit on a non standard port. My input config is:
input {
rabbitmq {
host => "10.241.160.103"
port => 30130
queue => "gcprice-k-queue"
user => "kuoni"
password => "Travel123"
codec => "plain"
}
}
The instance seems to start normally, however it balks at connecting to RabbitMQ on localhost:5672. Any idea where it's getting this definition from? He's a log extract (snipped part of it to beat the post character limit)
Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50}
2020-03-04 10:52:19,347 Converge PipelineAction::Create WARN The Logger slowlog.logstash.codecs.plain was created with the message factory org.apache.logging.log4j.spi.MessageFactory2Adapter@380b4b05 and is now requested with a null message factory (defaults to org.logstash.log.LogstashMessageFactory), which may create log events with unexpected formatting.
2020-03-04 10:52:20,019 Converge PipelineAction::Create WARN The Logger slowlog.logstash.filters.mutate was created with the message factory org.apache.logging.log4j.spi.MessageFactory2Adapter@4ad7c895 and is now requested with a null message factory (defaults to org.logstash.log.LogstashMessageFactory), which may create log events with unexpected formatting.
2020-03-04 10:52:20,106 Converge PipelineAction::Create WARN The Logger slowlog.logstash.outputs.elasticsearch was created with the message factory org.apache.logging.log4j.spi.MessageFactory2Adapter@532a8ad7 and is now requested with a null message factory (defaults to org.logstash.log.LogstashMessageFactory), which may create log events with unexpected formatting.
2020-03-04 10:52:20,112 Converge PipelineAction::Create WARN The Logger slowlog.logstash.codecs.plain was created with the message factory org.apache.logging.log4j.spi.MessageFactory2Adapter@380b4b05 and is now requested with a null message factory (defaults to org.logstash.log.LogstashMessageFactory), which may create log events with unexpected formatting.
[2020-03-04T10:52:20,210][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2020-03-04T10:52:20,421][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://10.241.162.203:9200/, http://10.241.162.205:9200/, http://10.241.162.206:9200/]}}
[2020-03-04T10:52:20,421][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://10.241.162.203:9200/, http://10.241.162.205:9200/, http://10.241.162.206:9200/]}}
[2020-03-04T10:52:20,432][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://10.241.162.203:9200/, :path=>"/"}
[2020-03-04T10:52:20,504][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://10.241.162.203:9200/, :path=>"/"}
[2020-03-04T10:52:21,311][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://10.241.162.203:9200/"}
[2020-03-04T10:52:21,312][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://10.241.162.203:9200/"}
[2020-03-04T10:52:21,434][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2020-03-04T10:52:21,434][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2020-03-04T10:52:21,435][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}
[2020-03-04T10:52:21,505][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}
[2020-03-04T10:52:21,505][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://10.241.162.205:9200/, :path=>"/"}
[2020-03-04T10:52:21,506][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://10.241.162.205:9200/, :path=>"/"}
[2020-03-04T10:52:21,510][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://10.241.162.205:9200/"}
[2020-03-04T10:52:21,512][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://10.241.162.205:9200/"}
[2020-03-04T10:52:21,519][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://10.241.162.206:9200/, :path=>"/"}
[2020-03-04T10:52:21,521][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://10.241.162.206:9200/"}
[2020-03-04T10:52:21,521][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://10.241.162.206:9200/, :path=>"/"}
[2020-03-04T10:52:21,523][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://10.241.162.206:9200/"}
[2020-03-04T10:52:21,615][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//10.241.162.203", "//10.241.162.205", "//10.241.162.206"]}
[2020-03-04T10:52:21,615][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//10.241.162.203", "//10.241.162.205", "//10.241.162.206"]}
[2020-03-04T10:52:21,706][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2020-03-04T10:52:21,807][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"default"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
2020-03-04 10:52:22,023 [.monitoring-logstash]>worker0 WARN The Logger slowlog.logstash.outputs.elasticsearch was created with the message factory org.apache.logging.log4j.spi.MessageFactory2Adapter@532a8ad7 and is now requested with a null message factory (defaults to org.logstash.log.LogstashMessageFactory), which may create log events with unexpected formatting.
2020-03-04 10:52:22,108 [.monitoring-logstash]>worker0 WARN The Logger slowlog.logstash.codecs.plain was created with the message factory org.apache.logging.log4j.spi.MessageFactory2Adapter@380b4b05 and is now requested with a null message factory (defaults to org.logstash.log.LogstashMessageFactory), which may create log events with unexpected formatting.
[2020-03-04T10:52:22,207][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://10.241.162.203:9200/, http://10.241.162.205:9200/, http://10.241.162.206:9200/]}}
[2020-03-04T10:52:22,208][INFO ][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://10.241.162.203:9200/, :path=>"/"}
[2020-03-04T10:52:22,211][WARN ][logstash.licensechecker.licensereader] Restored connection to ES instance {:url=>"http://10.241.162.203:9200/"}
[2020-03-04T10:52:22,328][INFO ][logstash.licensechecker.licensereader] ES Output version determined {:es_version=>6}
[2020-03-04T10:52:22,329][WARN ][logstash.licensechecker.licensereader] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>6}
[2020-03-04T10:52:22,329][INFO ][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://10.241.162.205:9200/, :path=>"/"}
[2020-03-04T10:52:22,334][WARN ][logstash.licensechecker.licensereader] Restored connection to ES instance {:url=>"http://10.241.162.205:9200/"}
[2020-03-04T10:52:22,408][INFO ][logstash.licensechecker.licensereader] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://10.241.162.206:9200/, :path=>"/"}
[2020-03-04T10:52:22,410][WARN ][logstash.licensechecker.licensereader] Restored connection to ES instance {:url=>"http://10.241.162.206:9200/"}
[2020-03-04T10:52:22,812][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x4c3f8376 run>"}
[2020-03-04T10:52:22,812][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>".monitoring-logstash", :thread=>"#<Thread:0x41cf5d39@/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:46 run>"}
[2020-03-04T10:52:23,026][INFO ][logstash.agent ] Pipelines running {:count=>2, :running_pipelines=>[:main, :".monitoring-logstash"], :non_running_pipelines=>}
[2020-03-04T10:52:23,112][INFO ][logstash.inputs.metrics ] Monitoring License OK
[2020-03-04T10:52:23,410][ERROR][logstash.inputs.rabbitmq ] RabbitMQ connection error, will retry. {:error_message=>"Connection to localhost:5672 refused", :exception=>"MarchHare::ConnectionRefused"}
[2020-03-04T10:52:24,419][ERROR][logstash.inputs.rabbitmq ] RabbitMQ connection error, will retry. {:error_message=>"Connection to localhost:5672 refused", :exception=>"MarchHare::ConnectionRefused"}
[2020-03-04T10:52:24,908][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2020-03-04T10:52:25,422][ERROR][logstash.inputs.rabbitmq ] RabbitMQ connection error, will retry. {:error_message=>"Connection to localhost:5672 refused", :exception=>"MarchHare::ConnectionRefused"}