Hi All,
First time post so please be gentle. I am pretty new to ELK but not to Linux.
I am running:
Server A - Remote Server
filebeat v 6.8.4
OS Ubuntu 18.04
Server B - ELK Server
Elastic Search 6.8.4
OS Ubuntu 18.04
Kibana
Filebeat v 6.8.4
Issue. I am unable to get remote logs into Elasticsearch via filebeat. I receive the following error. on Server A - Remote Server from the /var/log/filebeat/filebeat log.
"2019-11-19T15:26:23.362Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(https://my.elasticserver.mine:9200)): Get https://my.elasticserver.mine:9200: dial tcp my.elasticserver.mine:9200: connect: connection refused"
On Server B - ELK Server I can do a TCPDUMP and I can see the connection being made so the firewalling is all ok (I think)
17:31:10.526984 IP xx.xx.xx.xx.57740 > my.elasticserver.mine.9200: Flags [S], seq 3340582548, win 29200, options [mss 1460,sackOK,TS val 761458590 ecr 0,nop,wscale 7], length 0
17:31:10.527018 IP my.elasticserver.mine.9200 > xx.xx.xx.xx.57740: Flags [R.], seq 0, ack 3340582549, win 0, length 0
17:31:10.531121 IP xx.xx.xx.xx.57742 > my.elasticserver.mine.9200: Flags [S], seq 62320025, win 29200, options [mss 1460,sackOK,TS val 761458600 ecr 0,nop,wscale 7], length 0
17:31:10.531140 IP my.elasticserver.mine.9200 > xx.xx.xx.xx.57742: Flags [R.], seq 0, ack 62320026, win 0, length 0
My Filebeat.yml file from /etc/filebeat is as follows
###################### Filebeat Configuration Example #########################
filebeat.inputs:
- type: log
enabled: true
paths:- /var/log/syslog
- /var/log/*.log
- /var/log//.log
#- c:\programdata\elasticsearch\logs*
exclude_files: ['.gz$']
exclude_files: ['.?']
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 3
output.elasticsearch:
hosts: ["my.elasticserver.mine:9200"]
protocol: "https"
username: "somename"
password: some-password""
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
No Matter if i change passwords etc I still can not get my Server A - Remote Server logs into Server B- Elk Server.
Can anyone spot where I am going wrong? No, I am not shipping logs to logstash on Server B - ELK Server. I am shipping, or at least want to ship directly to Elastic search.
Here is the debug output from Server A - Remote Server
2019-11-19T17:16:50.036Z DEBUG [input] log/input.go:208 input states cleaned up. Before: 2, After: 2, Pending: 0
2019-11-19T17:16:50.036Z DEBUG [input] log/input.go:268 Exclude file: /var/log/syslog.2.gz
2019-11-19T17:16:50.036Z DEBUG [input] log/input.go:268 Exclude file: /var/log/syslog.3.gz
2019-11-19T17:16:50.036Z DEBUG [input] log/input.go:268 Exclude file: /var/log/syslog.4.gz
2019-11-19T17:16:50.036Z DEBUG [input] log/input.go:268 Exclude file: /var/log/syslog.5.gz
2019-11-19T17:16:50.036Z DEBUG [input] log/input.go:268 Exclude file: /var/log/syslog.6.gz
2019-11-19T17:16:50.036Z DEBUG [input] log/input.go:268 Exclude file: /var/log/syslog.7.gz
2019-11-19T17:16:50.036Z DEBUG [input] log/input.go:417 Check file for harvesting: /var/log/syslog
2019-11-19T17:16:50.036Z DEBUG [input] log/input.go:507 Update existing file for harvesting: /var/log/syslog, offset: 104857
2019-11-19T17:16:50.036Z DEBUG [input] log/input.go:559 Harvester for file is still running: /var/log/syslog
2019-11-19T17:16:50.036Z DEBUG [input] log/input.go:417 Check file for harvesting: /var/log/syslog.1
2019-11-19T17:16:50.037Z DEBUG [input] log/input.go:507 Update existing file for harvesting: /var/log/syslog.1, offset: 101093
2019-11-19T17:16:50.037Z DEBUG [input] log/input.go:559 Harvester for file is still running: /var/log/syslog.1
2019-11-19T17:16:50.037Z DEBUG [input] log/input.go:208 input states cleaned up. Before: 2, After: 2, Pending: 0
2019-11-19T17:16:58.201Z ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(https://my.elasticserver.mine:9200)): Get https://my.elasticserver.mine:9200: dial tcp my.elasticserver.mine:9200: connect: connection refused
2019-11-19T17:16:58.201Z INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(https://my.elasticserver.mine:9200)) with 15 reconnect attempt(s)
2019-11-19T17:16:58.201Z DEBUG [elasticsearch] elasticsearch/client.go:715 ES Ping(url=https://my.elasticserver.mine:9200)
2019-11-19T17:16:58.201Z INFO [publish] pipeline/retry.go:189 retryer: send unwait-signal to consumer
2019-11-19T17:16:58.201Z INFO [publish] pipeline/retry.go:191 done
2019-11-19T17:16:58.201Z INFO [publish] pipeline/retry.go:166 retryer: send wait signal to consumer
2019-11-19T17:16:58.201Z INFO [publish] pipeline/retry.go:168 done
2019-11-19T17:16:58.205Z DEBUG [elasticsearch] elasticsearch/client.go:719 Ping request failed with: Get https://77.68.95.148:9200: dial tcp my.elasticserver.mine:9200: connect: connection refused
2019-11-19T17:16:59.766Z DEBUG [input] input/input.go:152 Run input
2019-11-19T17:16:59.766Z DEBUG [input] log/input.go:187 Start next scan
2019-11-19T17:16:59.766Z DEBUG [input] log/input.go:268 Exclude file: /var/log/syslog
2019-11-19T17:16:59.767Z DEBUG [input] log/input.go:268 Exclude file: /var/log/alternatives.log
2019-11-19T17:16:59.767Z DEBUG [input] log/input.go:268 Exclude file: /var/log/auth.log
2019-11-19T17:16:59.767Z DEBUG [input] log/input.go:268 Exclude file: /var/log/bootstrap.log
2019-11-19T17:16:59.767Z DEBUG [input] log/input.go:268 Exclude file: /var/log/cloud-init-output.log
2019-11-19T17:16:59.767Z DEBUG [input] log/input.go:268 Exclude file: /var/log/cloud-init.log
2019-11-19T17:16:59.767Z DEBUG [input] log/input.go:268 Exclude file: /var/log/dpkg.log