Logstash is not listening on port 5044

Elastic Stack Logstash
1

I am using logstash version 5.3.0 . I have checked using netstat there is no port open of number 5044. I have checked log of logstash, found following error

An unexpected error occurred! {:error=># ArgumentError: Setting "http.host" must be a String. Received: ["10.10.10.21:5044"] (Array)>, :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/settings.rb:208:in validate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:384:invalidate'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:171:in set'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:61:inset_value'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:80:in merge'", "org/jruby/RubyHash.java:1342:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:80:in merge'", "/usr/share/logstash/logstash-core/lib/logstash/settings.rb:115:invalidate_all'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:210:in execute'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:67:inrun'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:183:in run'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/clamp-0.6.5/lib/clamp/command.rb:132:inrun'", "/usr/share/logstash/lib/bootstrap/environment.rb:71:in `(root)'"]}

I am clueless why this error occured.

Here is my non-commented logstash.yml

path.data: /var/lib/logstash
path.config: /etc/logstash/conf.d
http.host: ["10.10.10.21:5044"]
http.port: 9600-9700
path.logs: /var/log/logstash

1 Like
2

http.host: ["10.10.10.21:5044"]

I don't even know what this is supposed to mean. Why are you setting http.host at all? What are you trying to accomplish?

3

Thanks for the reply and happy sign up anniversary.
This is the ip address of machine where I installed logstash. I wanted to execute logstash on this ip and port.

4

I wanted to execute logstash on this ip and port.

That statement is still ambiguous, but it sounds like you should att one or more inputs to a file in /etc/logstash/conf.d instead of making changes in logstash.yml.

http.host controls the host where Logstash's monitoring API should listen. The value should not be an array and it should not contain a port number. The default value should be fine for you.

5

I set to http.host to default and observed logstash logs and there is no error found in log, but in output of netstat there entry like tcp6 0 0 :::5044 ::: LISTEN 46463/java*

I have installed filebeats on another node and observed this error in filebeat log as following

ERR Connecting error publishing events (retrying): read tcp 10.10.10.20:53376->10.10.10.21:5044: read: connection reset by peer

6

Please show your Logstash configuration (/etc/logstash/conf.d/*) and your Filebeat configuration.

7

There are three files in conf.d files.

02-beats-input.conf:

input {
beats {
port => 5044
}
}

10-syslog-filter.conf:

filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}

30-elasticsearch-output.conf:

output {
elasticsearch {
hosts => ["localhost"]
sniffing => true
manage_template => false
index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

filebeat.yml:

filebeat.prospectors:

  • input_type: log
    paths:
    • /var/log/auth.log
    • /var/log/syslog
      output.logstash:
      hosts: ["10.10.10.21:5044"]
      bulk_max_size: 2048
      ssl.certificate_authorities: ["/etc/filebeat/logstash.crt"]
      template.name: "filebeat"
      template.path: "filebeat.template.json"
      template.overwrite: false
8

You haven't enabled SSL on the Logstash side so don't attempt to use it from Filebeat. Remove or comment out the ssl.certificate_authorities line.

9

Thanks for the reply. I still found error in log file of filebeat as following

2017-05-05T10:10:15+05:30 INFO Registry file set to: /var/lib/filebeat/registry
2017-05-05T10:10:15+05:30 INFO Loading registrar data from /var/lib/filebeat/registry
2017-05-05T10:10:15+05:30 INFO States Loaded from registrar: 3
2017-05-05T10:10:15+05:30 INFO Loading Prospectors: 1
2017-05-05T10:10:15+05:30 INFO Prospector with previous states loaded: 3
2017-05-05T10:10:15+05:30 INFO Starting prospector of type: log; id: 881900914067917554
2017-05-05T10:10:15+05:30 INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2017-05-05T10:10:15+05:30 INFO Start sending events to output
2017-05-05T10:10:15+05:30 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017-05-05T10:10:15+05:30 INFO Starting Registrar
2017-05-05T10:10:25+05:30 INFO Harvester started for file: /var/log/auth.log
2017-05-05T10:10:25+05:30 INFO Harvester started for file: /var/log/syslog
2017-05-05T10:10:45+05:30 INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=2 filebeat.harvester.running=2 filebeat.harvester.started=2 libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.write_bytes=681 libbeat.publisher.published_events=11 publish.events=3 registrar.states.current=3 registrar.states.update=3 registrar.writes=1
2017-05-05T10:11:00+05:30 ERR Failed to publish events caused by: read tcp 10.10.10.20:56752->10.10.10.21:5044: i/o timeout
2017-05-05T10:11:00+05:30 INFO Error publishing events (retrying): read tcp 10.10.10.20:56752->10.10.10.21:5044: i/o timeout
2017-05-05T10:11:15+05:30 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=1 libbeat.logstash.publish.read_errors=1 libbeat.logstash.publish.write_bytes=504 libbeat.logstash.published_but_not_acked_events=11

10

And what's in the Logstash log around the same time?

11

I restart all services, here is logs of filebeats and logstash appearing on the same time

filebeat log:
2017-05-05T10:58:34+05:30 INFO Metrics logging every 30s
2017-05-05T10:58:34+05:30 INFO Starting Registrar
2017-05-05T10:58:34+05:30 INFO Start sending events to output
2017-05-05T10:58:34+05:30 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017-05-05T10:58:44+05:30 INFO Harvester started for file: /var/log/auth.log
2017-05-05T10:58:44+05:30 INFO Harvester started for file: /var/log/syslog
2017-05-05T10:58:49+05:30 ERR Connecting error publishing events (retrying): dial tcp 10.10.10.21:5044: getsockopt: connection refused
2017-05-05T10:58:50+05:30 ERR Connecting error publishing events (retrying): dial tcp 10.10.10.21:5044: getsockopt: connection refused
2017-05-05T10:58:52+05:30 ERR Connecting error publishing events (retrying): dial tcp 10.10.10.21:5044: getsockopt: connection refused
2017-05-05T10:58:56+05:30 ERR Connecting error publishing events (retrying): dial tcp 10.10.10.21:5044: getsockopt: connection refused
2017-05-05T10:59:04+05:30 INFO Non-zero metrics in the last 30s: filebeat.harvester.open_files=2 filebeat.harvester.running=2 filebeat.harvester.started=2 libbeat.publisher.published_events=36 publish.events=3 registrar.states.current=3 registrar.states.update=3 registrar.writes=1
2017-05-05T10:59:34+05:30 INFO Non-zero metrics in the last 30s: libbeat.logstash.call_count.PublishEvents=2 libbeat.logstash.publish.read_bytes=24 libbeat.logstash.publish.write_bytes=2909 libbeat.logstash.published_and_acked_events=38 libbeat.publisher.published_events=2 publish.events=40 registrar.states.update=40 registrar.writes=2

logstash log:
[2017-05-05T10:58:53,689][INFO ][logstash.inputs.beats ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2017-05-05T10:58:53,712][INFO ][logstash.pipeline ] Pipeline main started
[2017-05-05T10:58:53,752][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2017-05-05T10:58:58,261][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[http://localhost:9200/], :added=>[http://10.10.10.20:9200/, http://10.10.10.21:9200/, http://10.10.10.22:9200/]}}
[2017-05-05T10:58:58,261][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://10.10.10.20:9200/, :path=>"/"}
[2017-05-05T10:58:58,271][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0x4638e7d9 URL:http://10.10.10.20:9200/>}
[2017-05-05T10:58:58,272][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://10.10.10.21:9200/, :path=>"/"}
[2017-05-05T10:58:58,282][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0x5ae87c85 URL:http://10.10.10.21:9200/>}
[2017-05-05T10:58:58,289][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://10.10.10.22:9200/, :path=>"/"}
[2017-05-05T10:58:58,293][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>#<URI::HTTP:0x35861c98 URL:http://10.10.10.22:9200/>}

12

Judging by the final Filebeat messages things appear to be working fine. Between 2017-05-05T10:59:04+05:30 and 2017-05-05T10:59:34+05:30 it looks like it sent two events totalling 2909 bytes.

13

However there is still staus is Red on Kibana UI with following message:
ui settings Elasticsearch plugin is red
plugin:kibana@5.3.1 Ready
plugin:elasticsearch@5.3.1 Unable to connect to Elasticsearch at http://localhost:9200.
plugin:console@5.3.1 Ready
plugin:timelion@5.3.1 Ready

I have checked logs of elasticsearch also but didn't found any error.
Here is the output of curl GET http://10.10.10.21/_cat/plugins

<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx/1.10.0 (Ubuntu)</center>
</body>
</html>
14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.