Logs are not being pushed from beats to logstash

[rocky4bmw (rocky)]

Can anyone help me in fixing this issue

-> I configured Logstash 2.3.4 , Kibana 4.5.3 and ES 2.3.4 along with search guard bundle 2.3.4 on single master server.

-> I configured beats as per standard document on client server
paths:
- /var/log/auth.log
- /var/log/syslog
logstash:
# The Logstash hosts
hosts: ["ELK_server_private_IP:5044"]
tls:
# List of root certificates for HTTPS server verifications
certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]

-> When I try to restart beats (Filebeat / Topbeat ) I get below error message as below

topbeat -c topbeat.yml -e -v

2016/08/05 07:59:20.729491 geolite.go:24: INFO GeoIP disabled: No paths were set under output.geoip.paths
2016/08/05 07:59:20.730181 logstash.go:106: INFO Max Retries set to: 3
2016/08/05 07:59:20.730810 transport.go:125: ERR SSL client failed to connect with: dial tcp logstashserver ip:5044: getsockopt: connection refused
2016/08/05 07:59:20.730851 outputs.go:126: INFO Activated logstash as output plugin.
2016/08/05 07:59:20.731070 publish.go:288: INFO Publisher name: omsappbuild.softlayer.com
2016/08/05 07:59:20.731460 async.go:78: INFO Flush Interval set to: 1s
2016/08/05 07:59:20.731496 async.go:84: INFO Max Bulk Size set to: 2048
2016/08/05 07:59:20.731579 beat.go:147: INFO Init Beat: topbeat; Version: 1.2.3
2016/08/05 07:59:20.732334 beat.go:173: INFO topbeat sucessfully setup. Start running.
2016/08/05 07:59:31.732259 transport.go:125: ERR SSL client failed to connect with: dial tcp logstashserver ip:5044: getsockopt: connection refused
2016/08/05 07:59:31.732288 single.go:126: INFO Connecting error publishing events (retrying): dial tcp logstashserver ip:5044: getsockopt: connection refused
2016/08/05 07:59:31.732304 single.go:152: INFO send fail
2016/08/05 07:59:31.732315 single.go:159: INFO backoff retry: 1s
2016/08/05 07:59:32.732892 transport.go:125: ERR SSL client failed to connect with: dial tcp logstashserver ip:5044: getsockopt: connection refused
2016/08/05 07:59:32.732910 single.go:126: INFO Connecting error publishing events (retrying): dial tcp logstashserver ip:5044: getsockopt: connection refused
2016/08/05 07:59:32.732919 single.go:152: INFO send fail
2016/08/05 07:59:32.732928 single.go:159: INFO backoff retry: 2s
2016/08/05 07:59:34.733492 transport.go:125: ERR SSL client failed to connect with: dial tcp logstashserver ip:5044: getsockopt: connection refused
2016/08/05 07:59:34.733516 single.go:126: INFO Connecting error publishing events (retrying): dial tcp logstashserver ip:5044: getsockopt: connection refused
2016/08/05 07:59:34.733525 single.go:152: INFO send fail
2016/08/05 07:59:34.733536 single.go:159: INFO backoff retry: 4s
2016/08/05 07:59:38.734094 transport.go:125: ERR SSL client failed to connect with: dial tcp logstashserver ip:5044: getsockopt: connection refused
2016/08/05 07:59:38.734117 single.go:126: INFO Connecting error publishing events (retrying): dial tcp logstashserver ip:5044: getsockopt: connection refused
2016/08/05 07:59:41.732152 transport.go:125: ERR SSL client failed to connect with: dial tcp logstashserver ip:5044: getsockopt: connection refused
2016/08/05 07:59:41.732176 single.go:126: INFO Connecting error publishing events (retrying): dial tcp logstashserver ip:5044: getsockopt: connection refused
2016/08/05 07:59:41.732185 single.go:152: INFO send fail
2016/08/05 07:59:41.732195 single.go:159: INFO backoff retry: 1s
2016/08/05 07:59:42.732735 transport.go:125: ERR SSL client failed to connect with: dial tcp logstashserver ip:5044: getsockopt: connection refused
2016/08/05 07:59:42.732753 single.go:126: INFO Connecting error publishing events (retrying): dial tcp logstashserver ip:5044: getsockopt: connection refused
2016/08/05 07:59:42.732762 single.go:152: INFO send fail
2016/08/05 07:59:42.732771 single.go:159: INFO backoff retry: 2s
2016/08/05 07:59:44.733289 transport.go:125: ERR SSL client failed to connect with: dial tcp logstashserver ip:5044: getsockopt: connection refused
2016/08/05 07:59:44.733312 single.go:126: INFO Connecting error publishing events (retrying): dial tcp logstashserver ip:5044: getsockopt: connection refused
2016/08/05 07:59:44.733322 single.go:152: INFO send fail
2016/08/05 07:59:44.733332 single.go:159: INFO backoff retry: 4s
2016/08/05 07:59:48.733876 transport.go:125: ERR SSL client failed to connect with: dial tcp logstashserver ip:5044: getsockopt: connection refused
2016/08/05 07:59:48.733899 single.go:126: INFO Connecting error publishing events (retrying): dial tcp logstashserver ip:5044: getsockopt: connection refused
2016/08/05 07:59:51.732174 transport.go:125: ERR SSL client failed to connect with: dial tcp logstashserver ip:5044: getsockopt: connection refused
2016/08/05 07:59:51.732199 single.go:126: INFO Connecting error publishing events (retrying): dial tcp logstashserver ip:5044: getsockopt: connection refused

[rocky4bmw (rocky)]

Providing more info about logstash

-> Logstash configuration

input {
beats {
port => "5044"
ssl => true
ssl_certificate => "/etc/pki/tls/certs/xxx"
ssl_key => "/etc/pki/tls/private/xxx"
}
}
filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?:

%{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
user => "logstash"
password => "xxxx"
ssl => true
ssl_certificate_verification => true
truststore => "/home/test/test/elasticsearch-2.3.4-localhost/config/truststore.jks"
truststore_password => "xxxx"
manage_template => false
index => "%{[@metadata][beat][filebeat]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}

-> Logstash errors

{:timestamp=>"2016-08-05T02:56:20.048000-0500", :message=>"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid

certification path to requested target", :class=>"Manticore::ClientProtocolException", :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-

java/lib/manticore/response.rb:37:in initialize'", "org/jruby/RubyProc.java:281:incall'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-

java/lib/manticore/response.rb:79:in call'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:256:incall_once'",

"/opt/logstash/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:153:in `code'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-

transport-1.0.18/lib/elasticsearch/transport/transport/http/manticore.rb:84:in perform_request'", "org/jruby/RubyProc.java:281:incall'",

"/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/base.rb:257:in `perform_request'",

"/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'",

"/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.0.18/lib/elasticsearch/transport/client.rb:128:in `perform_request'",

"/opt/logstash/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.0.18/lib/elasticsearch/api/actions/bulk.rb:90:in `bulk'",

"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:53:in `non_threadsafe_bulk'",

"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk'",

"org/jruby/ext/thread/Mutex.java:149:in `synchronize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-

java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-

java/lib/logstash/outputs/elasticsearch/common.rb:172:in `safe_bulk'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-

java/lib/logstash/outputs/elasticsearch/common.rb:101:in `submit'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-

java/lib/logstash/outputs/elasticsearch/common.rb:86:in `retrying_submit'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-

java/lib/logstash/outputs/elasticsearch/common.rb:29:in multi_receive'", "org/jruby/RubyArray.java:1653:ineach_slice'",

"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:28:in `multi_receive'",

"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/output_delegator.rb:130:in `worker_multi_receive'",

"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/output_delegator.rb:114:in `multi_receive'",

"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:301:in output_batch'", "org/jruby/RubyHash.java:1342:ineach'",

"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:301:in `output_batch'",

"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:232:in `worker_loop'",

"/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:201:in `start_workers'"], :level=>:warn}

Note: I have installed Kibana and Logstash as root user and ES along with Searchguard bundle with non root user.

Can anyyou please guide me in fixing this issue?

[Sai_Birada]

It seems to be an ssl certificate issue. Are you able to establish ssl handshake using
curl -v --cacert logstash-forwarder.crt https://hostip:port ?.

[rocky4bmw (rocky)]

@Sai

Yes connection is not happening from beats to logstash server. It says connection refused

But I created open ssl certificates same way when I used nginx

cd /etc/pki/tls
sudo openssl req -config /etc/ssl/openssl.cnf -x509 -days 3650 -batch -nodes -newkey rsa:2048 -keyout private/logstash-forwarder.key -out certs/logstash-forwarder.crt

How can I fix this issue?

[rocky4bmw (rocky)]

I have followed same steps as per document in below link https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-centos-7
and it works fine with NGINX.

Is this the issue with logstash configuration?

some more logs from logstash on this issue

{:timestamp=>"2016-08-06T06:15:47.162000-0500", :message=>"Invalid setting for elasticsearch output plugin:\n\n output {\n elasticsearch {\n # This setting must be a path\n # File does not exist or cannot be opened /home/test/test/elasticsearch-2.3.4-localhost/config/truststore.jks\n truststore => "/home/test/test/elasticsearch-2.3.4-localhost/config/truststore.jks"\n ...\n }\n }", :level=>:error}
{:timestamp=>"2016-08-06T06:15:47.166000-0500", :message=>"Pipeline aborted due to error", :exception=>#<LogStash::ConfigurationError: Something is wrong with your configuration.>, :backtrace=>["/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/config/mixin.rb:134:in config_init'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/outputs/base.rb:63:ininitialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/output_delegator.rb:74:in register'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:181:instart_workers'", "org/jruby/RubyArray.java:1613:in each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:181:instart_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/pipeline.rb:136:in run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.3.4-java/lib/logstash/agent.rb:473:instart_pipeline'"], :level=>:error}
{:timestamp=>"2016-08-06T06:15:50.174000-0500", :message=>"stopping pipeline", :id=>"main"}

[Sai_Birada]

Hi Rocky,
Initially I followed the same procedure to establish ssl over filebeat-logstash connection, around couple of months back it worked properly, But since a month I am seeing the same errors, in many of my instances. You can check that issue here. This method is using the self signed certificates for the connection, and it is just one side authentication ( not two side ). So firstly, this procedure is not completely safe, but this procedure should work. The errors you faced could be because of the ssl version or cipher mismatch between your filebeat and logstash boxes. The better way is to simply create your own ca, and use its certificates to establish the connection.