I am trying to send log files using Filebeat to logstash. Filebeat and logstash are on different servers so Filebeat ip is like this 100.100.100.100 and logstash is like 200.200.200.200
My filebeat conf:
output.logstash:
# The Logstash hosts
hosts: ["200.200.200.200:5044"]
I have same problem. will watch this thread for any work aoround.
I have same setup
on client side
2019-02-15T00:07:12.184+0800 INFO log/harvester.go:255 Harvester started for file: /tmp/sensors.log
2019-02-15T00:07:38.528+0800 INFO [monitoring] log/log.go:144 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":17020,"time":{"ms":6}},"total":{"ticks":35260,"time":{"ms":12},"value":35260},"user":{"ticks":18240,"time":{"ms":6}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":7},"info":{"ephemeral_id":"d70b72e6-c2de-4f3a-aff4-33b5e9b11b98","uptime":{"ms":158250031}},"memstats":{"gc_next":4293824,"memory_alloc":2154176,"memory_total":1610054080}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":1,"update":1},"writes":{"success":1,"total":1}},"system":{"load":{"1":5.54,"15":9.64,"5":4.98,"norm":{"1":0.2308,"15":0.4017,"5":0.2075}}}}}}
Not quite, in you logstash input filter you do not have to specify an ip address. It will listen on all interfaces it will find. Filebeat however, still needs to know where to send the data. In your logstash output you need to configure an ip the server is listening on and is reachable from where ever you run filebeat on.
Now I can able to harwest files with filebeat however, logstash is not listening filebeat. And after every harwest I am seeing something like this
2019-02-15T09:41:18.796+0200 ERROR pipeline/output.go:74 Failed to connect: dial tcp 200.200.200.200:5044: connect
ex: A connection attempt failed because the connected party did not properly respond after a period of time, or establis
hed connection failed because connected host has failed to respond.
Is the ip 200.200.200.200 the ip that logstash is listening on? From the output you show I highly doubt that...
Not sure what type of machine you are using (linux/unix or windows) but you need to get the actual ip or hostname of the box where logstash is running on and put that in the filebeat.yml file at the logstash output.
###################### Filebeat Configuration Example #########################
#=========================== Filebeat prospectors =============================
filebeat.prospectors:
# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.
- type: log
# Change to true to enable this prospector configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- C:\Logs\Kurumlar\*.log
#- c:\programdata\elasticsearch\logs\*
### Multiline options
multiline.pattern: '^\['
multiline.negate: true
multiline.match: after
#============================= Filebeat modules ===============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: true
# Period on which files under path should be checked for changes
reload.period: 10s
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 3
#index.codec: best_compression
#_source.enabled: false
#================================ General =====================================
#============================== Dashboards =====================================
#============================== Kibana =====================================
#============================= Elastic Cloud ==================================
#================================ Outputs =====================================
#-------------------------- Elasticsearch output ------------------------------
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["200.200.200.200:5044"]
#setup.template:
# name: "testLog"
# pattern: "testLog-*"
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.