I'm trying to get logs (not syslog) from different machine servers and want to send those logs to logstash to elasticsearch to kibana. but in kibana I'm getting only system logs not other logs that I'm expecting.
Kindly help asap.
Thanks in Advance
My filebeat.yml file is
#=========================== Filebeat inputs =============================
filebeat.inputs:
- type: "log"
enabled: false
# Paths that should be crawled and fetched. Glob based paths.
paths:
#- /var/log/*.log
- /home/admin/Documents/Infrrd/wildfly/standalone/log/*.log
#- c:\programdata\elasticsearch\logs\*
#exclude_lines: ['^DBG']
#include_lines: ['^ERR', '^WARN']
#exclude_files: ['.gz$']
#fields:
# level: debug
# review: 1
#multiline.pattern: ^\[
#multiline.negate: false
#multiline.match: after
#============================= Filebeat modules ===============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: true
# Period on which files under path should be checked for changes
reload.period: 10s
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 3
#index.codec: best_compression
#_source.enabled: false
#================================ Outputs =====================================
# Configure what output to use when sending the data collected by the beat.
#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
# Array of hosts to connect to.
#hosts: ["localhost:9200"]
# Optional protocol and basic auth credentials.
#protocol: "https"
#username: "elastic"
#password: "changeme"
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["localhost:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
> **Configuration file in logstash-conf.conf is is**
input {
beats {
port => 5044
}
}
filter {
}
output {
elasticsearch {
hosts => ["localhost:9200"]
manage_template => false
index => "index-%{+YYYY.MM.dd}"
}
}
Maybe I'm missing something but you have enabled: false in your config file.
Also, maybe you haven't activated any module. Use filebeat modules list to see enabled and disabled modules. If you want to activate any of them, just use filebeat modules enable [module]
First of all thank you so much for replying and appreciate your efforts.
I tried using enabled: trued and I enabled modules system, logstash, redis but still it's not working. Is parsing mandatory in logstash if we are using server logs insted of syslog?
The main thing is I'm new to ELK so I don't know where to set the path for different machines server logs in filebeat.
Logstash is not mandatory. What I'll try is to set console output with the paths you need, only to see that everything is working. You should see JSON events being printed in the console.
Input config should be as simple as this. I mean that you shouldn't need more to make it work in its simplest way.
Check permissions too. And also Filebeat won't re-process files that it has already read.
If this is not working, then Filebeat is not using the correct configuration file. Use -c [path-to-configuration-file] when launching Filebeat to set the correct one
do you have any idea that why 5044 doesn't have any ip? I'm passing hosts as localhost:5044 in filebeat.yml and it logstash configuration file input port as 5044.
I was getting connection refused error. It is just because I was using "OpenJDK". And I installed "Oracle JDK " now it's working fine with Oracle jdk with less of configuration.
Special thanks to @Mario_Castro for replaying and supporting.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.