Filebeat logs not displaying to Kibana

[Codedestiny1 (Jonathan Loop)]

Filebeat YAML configuration files automated by Ansible playbook have been installed on two virtual machines (Web1 & Web2). I can SSH into them and find filebeat systemctl service is running (active). But no log data is displaying on Kibana.

Strangely though the same configuration for metric beat is displaying system metrics through Kibana. (Also Web1 & 2). I don't understand how metric data is coming through but system log data is not.

[warkolm (Mark Walkom)]

Welcome to our community!

What do your Filebeat logs show?
What version are you on?
What does your config look like?

[Codedestiny1 (Jonathan Loop)]

Hi Warkolm,

I havent tried to view the filebeat logs yet in the webservers. Which directory should I cd into and log file should I cat to provide that data? Also for additional reference, The two webservers are running DVWA and DVWA wasn't running correctly until i disabled Apache 2 on both webservers, not sure if that would be interfering with the kibana display for filebeat but wanted to let you know.

Filebeat version: 7.4.0-amd54.deb

Filebeat playbook:

---
- name: installing and launching filebeat
  hosts: webservers
  become: yes
  tasks:

    - name: download filebeat deb
      command: curl -L -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.4.0-amd64.deb

    - name: install filebeat deb
      command: dpkg -i filebeat-7.4.0-amd64.deb

    - name: drop in filebeat.yml
      copy:
        src: /etc/filebeat/filebeat-config.yml
        dest: /etc/filebeat/filebeat.yml

    - name: enable and configure system module
      command: filebeat modules enable system

    - name: setup filebeat
      command: filebeat setup

    - name: start filebeat service
      command: service filebeat start

    - name: enable service filebeat on boot
      systemd:
         name: filebeat
         enabled: yes

Host Configuration:

# This is the default ansible 'hosts' file.
#
# It should live in /etc/ansible/hosts
#
#   - Comments begin with the '#' character
#   - Blank lines are ignored
#   - Groups of hosts are delimited by [header] elements
#   - You can enter hostnames or ip addresses
#   - A hostname/ip can be a member of multiple groups

# Ex 1: Ungrouped hosts, specify before any group headers.

#green.example.com
#blue.example.com
#192.168.100.1
#192.168.100.10

# Ex 2: A collection of hosts belonging to the 'webservers' group

[webservers]
#alpha.example.org
#beta.example.org
#192.168.1.100
#192.168.1.110
10.0.0.7 ansible_python_interpreter=/usr/bin/python3
10.0.0.8 ansible_python_interpreter=/usr/bin/python3

[elk]
10.1.0.5 ansible_python_interpreter=/usr/bin/python3

# If you have multiple hosts following a pattern you can specify
# them like this:

#www[001:006].example.com

# Ex 3: A collection of database servers in the 'dbservers' group

#[dbservers]
#
#db01.intranet.mydomain.net
#db02.intranet.mydomain.net
#10.25.1.56
#10.25.1.57

# Here's another example of host ranges, this time there are no
# leading 0s:

#db-[99:101]-node.example.com

[Codedestiny1 (Jonathan Loop)]

Filebeat Configuration File:

#-------------------------- Elasticsearch output -------------------------------
output.elasticsearch:

  # Boolean flag to enable or disable the output module.
  #enabled: true

  # Array of hosts to connect to.
  # Scheme and port can be left out and will be set to the default (http and 9200)
  # In case you specify and additional path, the scheme is required: http://localhost:9200/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
  hosts: ["10.1.0.5:9200"]


#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
  host: "10.1.0.5:5601"
  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  #host: "localhost:5601"

[warkolm (Mark Walkom)]

Please format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you

[Codedestiny1 (Jonathan Loop)]

Thanks for the instructions Warkolm. I hope that reads better.

[warkolm (Mark Walkom)]

Great, thank you!

@Codedestiny1

I havent tried to view the filebeat logs yet in the webservers. Which directory should I cd into and log file should I cat to provide that data?

Probably /var/log/filebeat/filebeat.log.