Logstash is not listening port 5044

Elastic Stack Logstash
1

Hi,
Am new to Logstash, here my issue is logstash is not listening at port 5044,
here is my filebeat configuration output.logstash configured as following

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

and logs of my Filebeat service

● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/usr/lib/systemd/system/filebeat.service; disabled; vendor preset: disabled)
   Active: active (running) since Thu 2022-04-14 06:20:28 UTC; 4 days ago
     Docs: https://www.elastic.co/beats/filebeat
 Main PID: 6105 (filebeat)
   CGroup: /system.slice/filebeat.service
           └─6105 /usr/share/filebeat/bin/filebeat --environment systemd -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/lib/filebeat --path.logs /var/log/filebeat

Apr 18 12:03:18 logstash filebeat[6105]: 2022-04-18T12:03:18.671Z        INFO        [file_watcher]        filestream/fswatch.go:137        Start next scan
Apr 18 12:03:28 logstash filebeat[6105]: 2022-04-18T12:03:28.670Z        INFO        [monitoring]        log/log.go:184        Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":54900,"time":{"ms":10}},"total":{"ticks":198890,"time":{"ms":12},"value":198890},"user":{"ticks":143990,"time":{"ms":2}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":14},"info":{"ephemeral_id":"4063694c-ccfd-4233-9039-43986da0af70","uptime":{"ms":366180041},"version":"7.16.2"},"memstats":{"gc_next":32110672,"memory_alloc":17993256,"memory_total":7595562344,"rss":99135488},"runtime":{"goroutines":41}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0}},"pipeline":{"clients":5,"events":{"active":2282}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":3.57,"15":3.51,"5":3.46,"norm":{"1":0.8925,"15":0.8775,"5":0.865}}}}}}
Apr 18 12:03:28 logstash filebeat[6105]: 2022-04-18T12:03:28.670Z        INFO        [file_watcher]        filestream/fswatch.go:137        Start next scan
Apr 18 12:03:36 logstash filebeat[6105]: 2022-04-18T12:03:36.935Z        ERROR        [publisher_pipeline_output]        pipeline/output.go:154        Failed to connect to backoff(async(tcp://localhost:5044)): dial tcp 127.0.0.1:5044: connect: connection refused
Apr 18 12:03:36 logstash filebeat[6105]: 2022-04-18T12:03:36.935Z        INFO        [publisher_pipeline_output]        pipeline/output.go:145        Attempting to reconnect to backoff(async(tcp://localhost:5044)) with 31 reconnect attempt(s)
Apr 18 12:03:36 logstash filebeat[6105]: 2022-04-18T12:03:36.936Z        INFO        [publisher]        pipeline/retry.go:219        retryer: send unwait signal to consumer

and logs of Logstash service

● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2022-04-18 12:37:20 UTC; 9s ago
 Main PID: 6236 (java)
   CGroup: /system.slice/logstash.service
           └─6236 /usr/share/logstash/jdk/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -Djruby.regexp.interruptible=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -Dlog4j2.isThreadContextMapInheritable=true -cp /usr/share/logstash/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/share/logstash/logstash-core/lib/jars/checker-compat-qual-2.0.0.jar:/usr/share/logstash/logstash-core/lib/jars/commons-codec-1.14.jar:/usr/share/logstash/logstash-core/lib/jars/commons-compiler-3.1.0.jar:/usr/share/logstash/logstash-core/lib/jars/commons-logging-1.2.jar:/usr/share/logstash/logstash-core/lib/jars/error_prone_annotations-2.1.3.jar:/usr/share/logstash/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/guava-24.1.1-jre.jar:/usr/share/logstash/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-annotations-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-core-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-databind-2.9.10.8.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-yaml-2.9.10.jar:/usr/share/logstash/logstash-core/lib/jars/janino-3.1.0.jar:/usr/share/logstash/logstash-core/lib/jars/javassist-3.26.0-GA.jar:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.2.20.1.jar:/usr/share/logstash/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-1.2-api-2.17.0.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-api-2.17.0.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.17.0.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-jcl-2.17.0.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.17.0.jar:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/reflections-0.9.11.jar:/usr/share/logstash/logstash-core/lib/jars/slf4j-api-1.7.30.jar:/usr/share/logstash/logstash-core/lib/jars/snakeyaml-1.23.jar org.logstash.Logstash --path.settings /etc/logstash

Apr 18 12:37:20 logstash systemd[1]: Started logstash.
Apr 18 12:37:20 logstash logstash[6236]: Using bundled JDK: /usr/share/logstash/jdk
Apr 18 12:37:20 logstash logstash[6236]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

no listening port with 5044 is available in my system, can anyone helps me over this..

2

Hi @Meda_Akshay

Is there any firewall enabled on your server?
Have you tried using another port?

Best regards

3

Hi @Meda_Akshay welcome to the community.

Please share your logstash configuration

4

Hi @grfneto ,

No firewall is enabled and i have not tried with any other port.

according to logstash configuration, after starting logstash service, 5044 port need to be opened in my system but i could not see any listen port. hope something is missing in my logstash configuration

this is my logstash.conf file in conf.d/ diectory

#sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["https://vpc-elk-slmon-smsdlndsnfsdfksflk******.us-west-2.es.amazonaws.com:443"]
    ssl => true
    index => "logstash-%{[version]}-%{+YYYY.MM.dd}"
    user => "*******"
    password => "*******"
    ilm_enabled => false
  }
}
5

Hi @stephenb ,

this is my logstash.conf file in conf.d/ diectory

#sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["https://vpc-elk-askdjdklasdasdkklajkdasjdjd*****.us-west-2.es.amazonaws.com:443"]
    ssl => true
    index => "logstash-%{[version]}-%{+YYYY.MM.dd}"
    user => "****"
    password => "********"
    ilm_enabled => false
  }
}

following is in logstash.yml

path.data: /var/lib/logstash
api.http.host: 127.0.0.1
api.http.port: 5044
path.logs: /var/log/logstash
6

Why did you set these take them out that is not correct. Those settings are for monitoring the health of logstash.. and the should be left as default or set to port 9600

api.http.host: 127.0.0.1
api.http.port: 5044
7

let me check by removing those things

8

@stephenb same error i could see, no luck

9

How are you starting logstash?

Did you define your pipeline in the pipelines.yml? (Not needed if you put in /etc/logstash/conf.d if you installed as .deb. or .rpm) but still sometimes it is a good way to be precise

Have you looked in the logstash logs and see that it's actually starting the pipeline and opening the port?

You should see something like this in the logs...

[2022-04-18T07:27:22,324][INFO ][logstash.javapipeline    ][main] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>16, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>2000, "pipeline.sources"=>["/Users/sbrown/workspace/reference/configs/beats-logstash.conf"], :thread=>"#<Thread:0xbbf76bc run>"}
[2022-04-18T07:27:22,986][INFO ][logstash.javapipeline    ][main] Pipeline Java execution initialization time {"seconds"=>0.66}
[2022-04-18T07:27:23,003][INFO ][logstash.inputs.beats    ][main] Starting input listener {:address=>"0.0.0.0:5044"}. <!--- THIS ONE
[2022-04-18T07:27:23,013][INFO ][logstash.javapipeline    ][main] Pipeline started {"pipeline.id"=>"main"}
[2022-04-18T07:27:23,072][INFO ][logstash.agent           ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}

If you see that then try telnet... it should connect and wait

hyperion:tcp-syslog-logstash sbrown$ telnet 127.0.0.1 5044
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
10

yes i have defined pipelines.yml, following is my configuration

- pipeline.id: main
  path.config: "/etc/logstash/conf.d/logstash.conf"

with "logstash.conf" filename i defined the configuration in conf.d/

in logstash logs, as i defined above i couldn't see anything as like you sent, just showing me as "started logstash" and getting restarted for every 15 seconds

11

Where are you looking at the logs... you need to find the detailed logstash logs... that will tell you more ... not just the status from systemctl

Once you find the startup logs... share them

The logs should be in on unix

/var/log/logstash

example
tail -f /var/log/logstash/logstash-plain.log

OR you can start from the command line you will need to use the -f flag to point directly to you .conf file and you will also need to set the --path.settings SETTINGS_DIR this is a good way to debug as the logs will come to the console

12 
[2022-04-18T14:52:41,946][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2022-04-18T14:52:41,953][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.16.2", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[2022-04-18T14:52:42,914][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[2022-04-18T14:52:43,602][INFO ][org.reflections.Reflections] Reflections took 84 ms to scan 1 urls, producing 119 keys and 417 values 
[2022-04-18T14:52:44,532][INFO ][logstash.outputs.elasticsearch][main] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://vpc-elk-slmon-2u3vdecy6jc5ps5xstg7zkmgci.us-west-2.es.amazonaws.com:443"]}
[2022-04-18T14:52:44,808][INFO ][logstash.outputs.elasticsearch][main] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[https://slmon:xxxxxx@vpc-elk-slmon-2u3vdecy6jc5ps5xstg7zkmgci.us-west-2.es.amazonaws.com:443/]}}
[2022-04-18T14:52:45,158][ERROR][logstash.javapipeline    ][main] Pipeline error {:pipeline_id=>"main", :exception=>#<LogStash::ConfigurationError: Could not connect to a compatible version of Elasticsearch>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:247:in `block in healthcheck!'", "org/jruby/RubyHash.java:1415:in `each'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:240:in `healthcheck!'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:374:in `update_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:89:in `update_initial_urls'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:83:in `start'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:359:in `build_pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:63:in `initialize'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:106:in `create_http_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch/http_client_builder.rb:102:in `build'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/plugin_mixins/elasticsearch/common.rb:34:in `build_client'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-11.2.3-java/lib/logstash/outputs/elasticsearch.rb:275:in `register'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:131:in `register'", "org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:68:in `register'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:232:in `block in register_plugins'", "org/jruby/RubyArray.java:1821:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:231:in `register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:589:in `maybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:244:in `start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:189:in `run'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:141:in `block in start'"], "pipeline.sources"=>["/etc/logstash/conf.d/logstash.conf"], :thread=>"#<Thread:0x5714dd4b run>"}
[2022-04-18T14:52:45,162][INFO ][logstash.javapipeline    ][main] Pipeline terminated {"pipeline.id"=>"main"}
[2022-04-18T14:52:45,178][ERROR][logstash.agent           ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create<main>, action_result: false", :backtrace=>nil}
[2022-04-18T14:52:45,252][INFO ][logstash.runner          ] Logstash shut down.
[2022-04-18T14:52:45,260][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
	at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
	at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]

shows me an error "Could not connect to a compatible version of Elasticsearch"

13

Ok it is failing to connect to Elasticsearch...

Are you trying to connect to AWS Elasticsearch or Elastic Cloud (Official Elasticsearch)? Looks like the AWS / Opesearch version?

What version of Logstash are you running? This is probably the problem you should probably look at

And this forum can not really help with OpenSearch as that is an entire different product / codebase.

14

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

15

7.16.2 - logstash version and 7.10 - aws Elasticsearch version

16

I would revert to logstash 7.10 and / or use the opensearch plugin I showed above... sorry I don't know much about that plugin not sure if you can use logstash 7.16.2 + opensearch plugin you will need to try it I do not know anything about it

17

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

18

thanks a lot for your help.. :slight_smile:

19

yes, it is working with 7.10 thank you

20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.