Filebeat pushs syslog logs to elasticsearch through Logstash

Hi everyone!

I'm trying to push syslog logs to elasticsearch by using Filebeat and Logstash. How should my configuration files look like?

#=========================== Filebeat inputs =============================


# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: false

  # Paths that should be crawled and fetched. Glob based paths.
- /var/log/*.log
#- c:\programdata\elasticsearch\logs\*

#----------------------------- Logstash output --------------------------------
  # The Logstash hosts
  hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

  - add_host_metadata: ~
  - add_cloud_metadata: ~

My logstash conf.d file looks like

input {
  beats {
port => 5044

filter {
  grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
  geoip { source => "clientip" }

output {
  elasticsearch {
   hosts => ["localhost:9200"]
   index => "geoip3"

Everything looks like it's working but I cannot see any data coming to elasticsearch :frowning:

I'd like to use logger command and see the log message in elasticsearch via Kibana

Thanks for any help ))

To debug any issue, it is better to have a simple use case.
You should first make sure Filebeat is reading the files and data from Filebeat is reaching Logstash.

Ensure Filebeat is reading files : --
1) Add logging (refer filebeat_reference.yml) for Filebeat.

#================================ Logging ======================================
# There are four options for the log output: file, stderr, syslog, eventlog
# The file output is the default.

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: info

# Enable debug output for selected components. To enable all selectors use ["*"]
# Other available selectors are "beat", "publish", "service"
# Multiple selectors can be chained.
#logging.selectors: [ ]

# Send all logging output to stderr. The default is false.
#logging.to_stderr: false

# Send all logging output to syslog. The default is false.
#logging.to_syslog: false

# Send all logging output to Windows Event Logs. The default is false.
#logging.to_eventlog: false

# If enabled, Filebeat periodically logs its internal metrics that have changed
# in the last period. For each metric that changed, the delta from the value at
# the beginning of the period is logged. Also, the total values for
# all non-zero internal metrics are logged on shutdown. The default is true.
#logging.metrics.enabled: true

# The period after which to log the internal metrics. The default is 30s.
logging.metrics.period: 30s

# Logging to rotating files. Set logging.to_files to false to disable logging to
# files.
logging.to_files: true
  # Configure the path where the logs are written. The default is the logs directory
  # under the home path (the binary location).
  path: /var/log/filebeat

  # The name of the files where the logs are written to.
  name: filebeat

  # Configure log file size limit. If limit is reached, log file will be
  # automatically rotated
  rotateeverybytes: 10485760 # = 10MB

  # Number of rotated log files to keep. Oldest files will be deleted first.
  keepfiles: 7

  # The permissions mask to apply when rotating log files. The default value is 0600.
  # Must be a valid Unix-style file permissions mask expressed in octal notation.
  permissions: 0600

  # Enable log file rotation on time intervals in addition to size-based rotation.
  # Intervals must be at least 1s. Values of 1m, 1h, 24h, 7*24h, 30*24h, and 365*24h
  # are boundary-aligned with minutes, hours, days, weeks, months, and years as
  # reported by the local system clock. All other intervals are calculated from the
  # Unix epoch. Defaults to disabled.
  interval: 0

  # Rotate existing logs on startup rather than appending to the existing
  # file. Defaults to true.
  # rotateonstartup: true

# Set to true to log messages in JSON format.
logging.json: false
  1. Start filebeat by passing your config file explicitly
    /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml (alter this command based on your local machine settings)

  2. After starting Filebeat, add a new file in /var/log folder

  3. Check filebeat log (/var/log/filebeat/filebeat) that the harvester started for new file

Logstash receiving data from Filebeat :-

Instead of sending Logstash output to ES please direct it to stdout.


input {
  beats {
    port => 5044

output {
	stdout { 
	     codec => rubydebug { } 

After I added Logging part that you suggested into filebeat.yml I cannot even start filebeat. It gives me following message when I try to find out status of filebeat

● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2019-10-25 07:06:57 UTC; 5s ago
  Process: 2313 ExecStart=/usr/share/filebeat/bin/filebeat $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS (code=exited, status=1/FAILURE)
 Main PID: 2313 (code=exited, status=1/FAILURE)

Oct 25 07:06:57 bak2 systemd[1]: filebeat.service: Service hold-off time over, scheduling restart.
Oct 25 07:06:57 bak2 systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 5.
Oct 25 07:06:57 bak2 systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
Oct 25 07:06:57 bak2 systemd[1]: filebeat.service: Start request repeated too quickly.
Oct 25 07:06:57 bak2 systemd[1]: filebeat.service: Failed with result 'exit-code'.
Oct 25 07:06:57 bak2 systemd[1]: Failed to start Filebeat sends log files to Logstash or directly to Elasticsearch..

YAML files can be very sensitive to space and indentation.
Please check filebeat_reference.yml and copy the logging part into filebeat.yml
Make changes to settings if required and then try starting Filebeat again

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.