[Solved] Filebeat 7.0.1 has no data on [Filebeat System] Syslog dashboard ECS

Hello team,

I have an ELK stack running on 7.0.1 and i have trouble displaying data on the [Filebeat System] Syslog dashboard ECS but i can see logs on the discover panel.
I have configured a linuxclient to send system logs using filebeat to elasticsearch nodes straight. I am using the system module from filebeat.

My linuxclient is running RHEL 7.0 OS and i am trying to display the important logs on kibana.

On my linuxclient:

system.yml

- module: system
  # Syslog
  syslog:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/dmesg"]

    # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
    var.convert_timezone: true

  # Authorization logs
  auth:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["/var/log/secure"]

    # Convert the timestamp to UTC. Requires Elasticsearch >= 6.1.
    var.convert_timezone: true
[root@linuxclient modules.d]#

filebeat.yml

#=========================== Filebeat inputs =============================

filebeat.inputs:

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/messages
    - /var/log/cron
    - /var/log/secure
    #- c:\programdata\elasticsearch\logs\*

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ['^DBG']
  exclude_lines: ['.*monitoring.*']

#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: true

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  host: "kibana:5601"


#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["esnode1:9200", "esnode2:9200", "esnode3:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
#output.logstash:
  # The Logstash hosts
  # hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

#================================ Logging =====================================

#logging.level: debug

#logging.selectors: ["*"]

#============================== Xpack Monitoring

#xpack.monitoring.enabled: false

#xpack.monitoring.elasticsearch:

# Migration 

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true

setup.ilm.enabled: auto
setup.ilm.rollover_alias: "filebeat-linuxclient"
setup.ilm.pattern: "{now/d}-000001"
setup.template.overwrite: true

i have deleted some commented configs due to exceeding word count on this post
Any idea what could be the problem?

Solved this issue.
Apparently, my mistake is that i have loaded "- /var/log/messages" in my prospector settings and i should have place this into the system.yml module.
After that, the dashboard is displaying correctly.

Great! Thanks for posting it here! @TsuWeiQuan

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.