Logstash doesn’t seem to open new port for new pipeline. Filebeat error “no connection could be made because the target machine actively refused it”

Elastic Stack Logstash
1

Hi community,
I’ve been looking for some help regarding to the following scenario.

The ELK in use it’s running over RKE in Red Hat.

Logstash doesn’t seem to open a new port for new pipeline. Filebeat error “no connection could be made because the target machine actively refused it”

Considerations: iptables and firewall are running.

2

What do you have in Logstash logs? What is the input configuration?

Please share the logs from Logstash, try to restart it to get fresh logs and share them, look for the port of the new pipeline in the logs, it will have a message about it or an error if it cannot bind to the port.

Also, did you check if the iptables and firewall are correctly configured to allow traffic from your filebeat to your logstash server?

3

Thanks for the reply. Yes I already accept the port on ip tables and firewall.

Config .conf on Logstash:

logserverwindows.conf
input {

   beats {

       port => 5043

       ssl => false

                }

}

output {

elasticsearch {

index => "indextesting-%{[agent.version]}-%{+yyyy.MM.dd}"



user => "accesselastic"



password => "xxxxxxxxxxxxxx"



hosts => ["http://elasticsearchnodo1.svc.cluster.local:9200"]



ssl_certificate_verification => false

}

}

Logs results after refresh pods:

Blockquote
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Sending Logstash logs to /usr/share/logstash/logs which is now configured via log4j2.properties
[2024-06-26T23:41:15,636][INFO ][logstash.runner ] Log4j configuration path used is: /usr/share/logstash/config/log4j2.properties
[2024-06-26T23:41:15,657][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.17.3", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.14.1+1 on 11.0.14.1+1 +indy +jit [linux-x86_64]"}
[2024-06-26T23:41:15,660][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Dls.cgroup.cpuacct.path.override=/, -Dls.cgroup.cpu.path.override=/, -Xmx2g, -Xms2g]
[2024-06-26T23:41:15,711][INFO ][logstash.settings ] Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[2024-06-26T23:41:15,731][INFO ][logstash.settings ] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[2024-06-26T23:41:16,404][INFO ][logstash.agent ] No persistent UUID file found. Generating new UUID {:uuid=>"xxxxxxxx-xxxx-xxxx-xxxx-78c846474f0d", :path=>"/usr/share/logstash/data/uuid"}
[2024-06-26T23:41:18,459][WARN ][deprecation.logstash.monitoringextension.pipelineregisterhook] Internal collectors option for Logstash monitoring is deprecated and may be removed in a future release.
Please configure Metricbeat to monitor Logstash. Documentation can be found at:
Collect Logstash monitoring data with Metricbeat | Logstash Reference [8.14] | Elastic
[2024-06-26T23:41:19,122][WARN ][deprecation.logstash.codecs.plain] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:19,230][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:19,896][INFO ][logstash.licensechecker.licensereader] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://logstash_sys:xxxxxx@elastic-master:9200/]}}
[2024-06-26T23:41:20,325][WARN ][logstash.licensechecker.licensereader] Restored connection to ES instance {:url=>"http://logstash_sys:xxxxxx@elastic-master:9200/"}
[2024-06-26T23:41:20,356][INFO ][logstash.licensechecker.licensereader] Elasticsearch version determined (7.17.3) {:es_version=>7}
[2024-06-26T23:41:20,360][WARN ][logstash.licensechecker.licensereader] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[2024-06-26T23:41:20,550][INFO ][logstash.monitoring.internalpipelinesource] Monitoring License OK
[2024-06-26T23:41:20,553][INFO ][logstash.monitoring.internalpipelinesource] Validated license for monitoring. Enabling monitoring pipeline.
[2024-06-26T23:41:20,998][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2024-06-26T23:41:29,860][INFO ][org.reflections.Reflections] Reflections took 508 ms to scan 1 urls, producing 119 keys and 419 values
[2024-06-26T23:41:34,954][WARN ][deprecation.logstash.codecs.plain] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:34,954][WARN ][deprecation.logstash.codecs.plain] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:34,955][WARN ][deprecation.logstash.codecs.plain] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:34,967][WARN ][deprecation.logstash.codecs.plain] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:34,968][WARN ][deprecation.logstash.codecs.plain] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:34,982][WARN ][deprecation.logstash.codecs.plain] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:34,996][WARN ][deprecation.logstash.inputs.beats] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:34,997][WARN ][deprecation.logstash.inputs.beats] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:35,005][WARN ][deprecation.logstash.inputs.beats] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:35,007][WARN ][deprecation.logstash.inputs.beats] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:35,052][WARN ][deprecation.logstash.inputs.beats] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:35,366][WARN ][deprecation.logstash.codecs.plain] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:35,375][WARN ][deprecation.logstash.codecs.plain] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:35,460][WARN ][deprecation.logstash.codecs.plain] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:35,465][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:35,467][WARN ][deprecation.logstash.outputs.elasticsearchmonitoring] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:35,551][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:35,578][WARN ][deprecation.logstash.codecs.plain] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:35,662][WARN ][deprecation.logstash.filters.geoip] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:35,777][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:35,803][WARN ][deprecation.logstash.codecs.plain] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:35,850][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearchMonitoring", :hosts=>["http://elastic-master:9200"]}
[2024-06-26T23:41:35,852][INFO ][logstash.outputs.elasticsearch][logserverwindows] New Elasticsearch output {:class=>"LogStash::Outputs::Elasticsearch", :hosts=>["http://elascticnodo1-monitoring.svc.cluster.local:9200"]}
[2024-06-26T23:41:36,056][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://logstash_sys:xxxxxx@elastic-master:9200/]}}
[2024-06-26T23:41:36,056][INFO ][logstash.outputs.elasticsearch][logserverwindows] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=>[http://userelastic:xxxxxx@elascticnodo1-monitoring.svc.cluster.local:9200/]}}
[2024-06-26T23:41:36,328][WARN ][deprecation.logstash.codecs.plain] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:36,361][WARN ][logstash.outputs.elasticsearch][logserverwindows] Restored connection to ES instance {:url=>"http://userelastic:xxxxxx@elascticnodo1-monitoring.svc.cluster.local:9200/"}
[2024-06-26T23:41:36,361][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Restored connection to ES instance {:url=>"http://logstash_sys:xxxxxx@elastic-master:9200/"}
[2024-06-26T23:41:36,386][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Elasticsearch version determined (7.17.3) {:es_version=>7}
[2024-06-26T23:41:36,386][INFO ][logstash.outputs.elasticsearch][logserverwindows] Elasticsearch version determined (7.17.3) {:es_version=>7}
[2024-06-26T23:41:36,387][WARN ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[2024-06-26T23:41:36,387][WARN ][logstash.outputs.elasticsearch][logserverwindows] Detected a 6.x and above cluster: the type event field won't be used to determine the document _type {:es_version=>7}
[2024-06-26T23:41:36,474][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:36,617][WARN ][deprecation.logstash.codecs.plain] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:36,669][INFO ][logstash.outputs.elasticsearchmonitoring][.monitoring-logstash] Config is not compliant with data streams. data_stream => auto resolved to false
[2024-06-26T23:41:36,674][INFO ][logstash.outputs.elasticsearch][logserverwindows] Config is not compliant with data streams. data_stream => auto resolved to false
[2024-06-26T23:41:36,688][INFO ][logstash.outputs.elasticsearch][logserverwindows] Config is not compliant with data streams. data_stream => auto resolved to false
[2024-06-26T23:41:36,751][WARN ][logstash.javapipeline ][.monitoring-logstash] 'pipeline.ordered' is enabled and is likely less efficient, consider disabling if preserving event order is not necessary
[2024-06-26T23:41:36,856][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:36,858][INFO ][logstash.outputs.elasticsearch][logserverwindows] Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[2024-06-26T23:41:37,244][INFO ][logstash.javapipeline ][.monitoring-logstash] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.workers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2, "pipeline.sources"=>["monitoring pipeline"], :thread=>"#<Thread:0x61cf0d9e@/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:129 run>"}
[2024-06-26T23:41:37,252][INFO ][logstash.javapipeline ][logserverwindows] Starting pipeline {:pipeline_id=>"logserverwindows", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/usr/share/logstash/pipeline/logserverwindows.conf"], :thread=>"#<Thread:0x3155b0ba@/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:54 run>"}
[2024-06-26T23:41:41,218][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline Java execution initialization time {"seconds"=>3.97}
[2024-06-26T23:41:41,569][INFO ][logstash.javapipeline ][.monitoring-logstash] Pipeline started {"pipeline.id"=>".monitoring-logstash"}
[2024-06-26T23:41:41,581][INFO ][logstash.javapipeline ][logserverwindows] Pipeline Java execution initialization time {"seconds"=>4.31}
[2024-06-26T23:41:41,852][INFO ][logstash.inputs.beats ][logserverwindows] Starting input listener {:address=>"0.0.0.0:5043"}
[2024-06-26T23:41:42,045][INFO ][logstash.javapipeline ][logserverwindows] Pipeline started {"pipeline.id"=>"logserverwindows"}
[2024-06-26T23:41:43,774][WARN ][deprecation.logstash.codecs.plain] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:43,802][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:43,818][WARN ][deprecation.logstash.codecs.plain] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:43,856][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:43,869][WARN ][deprecation.logstash.codecs.plain] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:43,885][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:43,899][WARN ][deprecation.logstash.codecs.plain] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:43,914][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:43,952][WARN ][deprecation.logstash.codecs.plain] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:43,969][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:43,982][WARN ][deprecation.logstash.codecs.plain] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:43,997][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:44,011][WARN ][deprecation.logstash.codecs.plain] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:44,052][WARN ][deprecation.logstash.outputs.elasticsearch] Relying on default value of pipeline.ecs_compatibility, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[2024-06-26T23:41:56,317][INFO ][logstash.agent ] Pipelines running {:count=>7, :running_pipelines=>[:".monitoring-logstash", :logserverwindows, :pipelin1, :pipelin2, :pipelin3, :pipelin4, :pipelin5], :non_running_pipelines=>}

4

This means that logstah is listening on port 5043, you can validate it with a netstat for example.

1 Like
5

Not running port on the 3 logstash nodes in use after netstat command.

6

Not need to validate, LS doesnt' lies. There is 7 pipelines:
Pipelines running {:count=>7, :running_pipelines=>[:".monitoring-logstash", :logserverwindows, :pipelin1, :pipelin2, :pipelin3, :pipelin4, :pipelin5]
Can be mess between input in .conf the same port or something.

[Mhvrke], please confirm:
telnet <logstashserver> 5043

If still not working, review other inputs in all .confS. You have copied only 1 single .conf