How to configure different machine servers log in filebeat and send it to logstash

Vinit_Kumar · January 2, 2019, 6:27am

Hi All,

I'm trying to get logs (not syslog) from different machine servers and want to send those logs to logstash to elasticsearch to kibana. but in kibana I'm getting only system logs not other logs that I'm expecting.
Kindly help asap.

Thanks in Advance

My filebeat.yml file is

#=========================== Filebeat inputs =============================

filebeat.inputs:

- type: "log"

  enabled: false

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    - /home/admin/Documents/Infrrd/wildfly/standalone/log/*.log
    #- c:\programdata\elasticsearch\logs\*

  #exclude_lines: ['^DBG']

  #include_lines: ['^ERR', '^WARN']

  #exclude_files: ['.gz$']

  #fields:
  #  level: debug
  #  review: 1

  #multiline.pattern: ^\[

  #multiline.negate: false

  #multiline.match: after

#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: true

  # Period on which files under path should be checked for changes
  reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"





> **Configuration file in logstash-conf.conf is is**

input {
  beats {
    port => 5044
  }
}
filter {
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    manage_template => false
    index => "index-%{+YYYY.MM.dd}"
  }
}

Mario_Castro · January 2, 2019, 11:08am

Hi @Vinit_Kumar

Maybe I'm missing something but you have enabled: false in your config file.

Also, maybe you haven't activated any module. Use filebeat modules list to see enabled and disabled modules. If you want to activate any of them, just use filebeat modules enable [module]

I hope it helps

Vinit_Kumar · January 2, 2019, 11:17am

Hi @Mario_Castro

First of all thank you so much for replying and appreciate your efforts.

I tried using enabled: trued and I enabled modules system, logstash, redis but still it's not working. Is parsing mandatory in logstash if we are using server logs insted of syslog?

The main thing is I'm new to ELK so I don't know where to set the path for different machines server logs in filebeat.

Mario_Castro · January 2, 2019, 11:59am

Logstash is not mandatory. What I'll try is to set console output with the paths you need, only to see that everything is working. You should see JSON events being printed in the console.

Input config should be as simple as this. I mean that you shouldn't need more to make it work in its simplest way.

Check permissions too. And also Filebeat won't re-process files that it has already read.

Vinit_Kumar · January 2, 2019, 2:48pm

I tried but it's not working.

Just now I installed ELK again from ELK official website documentation.
Do you know how to change the default input path of logs in filebeat.yml

default is /var/log/*.log
to
other file directory

Mario_Castro · January 2, 2019, 4:59pm

It's in the link above.

filebeat.inputs:
- type: log
  paths:
    - /var/log/messages
    - /var/log/*.log
    - /my/custom/path

If this is not working, then Filebeat is not using the correct configuration file. Use -c [path-to-configuration-file] when launching Filebeat to set the correct one

Vinit_Kumar · January 3, 2019, 4:36am

@Mario_Castro Thanks for replying again.

dmin@inflap167-HP-EliteBook-840-G2: sudo netstat -ntlp | grep LISTEN

tcp 0 0 0.0.0.0:8787 tcp 0 0 127.0.1.1:53 tcp 0 0 127.0.0.1:631 tcp 0 0 127.0.0.1:8443 tcp 0 0 127.0.0.1:5601 tcp 0 0 127.0.0.1:9990 tcp 0 0 0.0.0.0:27017 tcp 0 0 127.0.0.1:9001 tcp 0 0 127.0.0.1:3306 tcp 0 0 127.0.0.1:6379 tcp 0 0 127.0.0.1:8080 tcp6 0 0 :::5044 tcp6 0 0 127.0.0.1:9300 tcp6 0 0 :::8983 tcp6 0 0 ::1:631 tcp6 0 0 :::45657 tcp6 0 0 127.0.0.1:9600 tcp6 0 0 :::6627 tcp6 0 0 :::9092 tcp6 0 0 :::32965 tcp6 0 0 :::2181 tcp6 0 0 :::9898 tcp6 0 0 127.0.0.1:9200 0.0.0.0:* LISTEN 1914/java
0.0.0.0:* LISTEN 2653/dnsmasq
0.0.0.0:* LISTEN 1117/cupsd
0.0.0.0:* LISTEN 1914/java
0.0.0.0:* LISTEN 6643/node
0.0.0.0:* LISTEN 1914/java
0.0.0.0:* LISTEN 1788/mongod
0.0.0.0:* LISTEN 1548/python
0.0.0.0:* LISTEN 1551/mysqld
0.0.0.0:* LISTEN 1565/redis-server 1
0.0.0.0:* LISTEN 1914/java
:::* LISTEN 16234/java
:::* LISTEN 16122/java
:::* LISTEN 1918/java
:::* LISTEN 1117/cupsd
:::* LISTEN 1287/java
:::* LISTEN 16234/java
:::* LISTEN 1760/java
:::* LISTEN 1761/java
:::* LISTEN 1761/java
:::* LISTEN 1287/java
:::* LISTEN 1764/java
:::* LISTEN 16122/java

do you have any idea that why 5044 doesn't have any ip? I'm passing hosts as localhost:5044 in filebeat.yml and it logstash configuration file input port as 5044.

but in kibana still I'm getting custome logs.

but in filebeat.yml I'm passing inout as bellow..

filebeat.inputs:

type: log
enabled: true
paths:
- /var/log/custom-log.log
#- /home/admin/ELK/logs/*.log

Vinit_Kumar · January 7, 2019, 3:47pm

I was getting connection refused error. It is just because I was using "OpenJDK". And I installed "Oracle JDK " now it's working fine with Oracle jdk with less of configuration.

Special thanks to @Mario_Castro for replaying and supporting.

system · February 4, 2019, 3:47pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
How to configure server logs from different server using filebeat and send logs to logstash which installed on other server Beats filebeat	2	2297	March 30, 2022
Error in filebeat when sending logs to kibana Beats	15	2470	May 14, 2019
Filebeat, Kibana and ElasticSearch are configured, but not reading log files Beats filebeat	6	1491	August 14, 2020
ELK stack with filebeat not mapping well GeoIP data Logstash	6	1227	January 30, 2020
Cannot ship ssh logs from filebeat to logstash Beats filebeat	4	786	February 18, 2021

How to configure different machine servers log in filebeat and send it to logstash

Related topics