Filebeat pushs syslog logs to elasticsearch through Logstash

[Vladpov]

Hi everyone!

I'm trying to push syslog logs to elasticsearch by using Filebeat and Logstash. How should my configuration files look like?

#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: false

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
- /var/log/*.log
#- c:\programdata\elasticsearch\logs\*

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

My logstash conf.d file looks like

input {
  beats {
port => 5044
  }
}


filter {
  grok { match => { "message" => "%{COMBINEDAPACHELOG}" } }
  geoip { source => "clientip" }
}


output {
  elasticsearch {
   hosts => ["localhost:9200"]
   index => "geoip3"
  }
}

Everything looks like it's working but I cannot see any data coming to elasticsearch

I'd like to use logger command and see the log message in elasticsearch via Kibana

Thanks for any help ))

[kumarabhi (Abhishek)]

To debug any issue, it is better to have a simple use case.
You should first make sure Filebeat is reading the files and data from Filebeat is reaching Logstash.

Ensure Filebeat is reading files : --
-------------------------------------------------
1) Add logging (refer filebeat_reference.yml) for Filebeat.

#================================ Logging ======================================
# There are four options for the log output: file, stderr, syslog, eventlog
# The file output is the default.

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: info

# Enable debug output for selected components. To enable all selectors use ["*"]
# Other available selectors are "beat", "publish", "service"
# Multiple selectors can be chained.
#logging.selectors: [ ]

# Send all logging output to stderr. The default is false.
#logging.to_stderr: false

# Send all logging output to syslog. The default is false.
#logging.to_syslog: false

# Send all logging output to Windows Event Logs. The default is false.
#logging.to_eventlog: false

# If enabled, Filebeat periodically logs its internal metrics that have changed
# in the last period. For each metric that changed, the delta from the value at
# the beginning of the period is logged. Also, the total values for
# all non-zero internal metrics are logged on shutdown. The default is true.
#logging.metrics.enabled: true

# The period after which to log the internal metrics. The default is 30s.
logging.metrics.period: 30s

# Logging to rotating files. Set logging.to_files to false to disable logging to
# files.
logging.to_files: true
logging.files:
  # Configure the path where the logs are written. The default is the logs directory
  # under the home path (the binary location).
  path: /var/log/filebeat

  # The name of the files where the logs are written to.
  name: filebeat

  # Configure log file size limit. If limit is reached, log file will be
  # automatically rotated
  rotateeverybytes: 10485760 # = 10MB

  # Number of rotated log files to keep. Oldest files will be deleted first.
  keepfiles: 7

  # The permissions mask to apply when rotating log files. The default value is 0600.
  # Must be a valid Unix-style file permissions mask expressed in octal notation.
  permissions: 0600

  # Enable log file rotation on time intervals in addition to size-based rotation.
  # Intervals must be at least 1s. Values of 1m, 1h, 24h, 7*24h, 30*24h, and 365*24h
  # are boundary-aligned with minutes, hours, days, weeks, months, and years as
  # reported by the local system clock. All other intervals are calculated from the
  # Unix epoch. Defaults to disabled.
  interval: 0

  # Rotate existing logs on startup rather than appending to the existing
  # file. Defaults to true.
  # rotateonstartup: true

# Set to true to log messages in JSON format.
logging.json: false

2. Start filebeat by passing your config file explicitly
   /usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat.yml (alter this command based on your local machine settings)

3. After starting Filebeat, add a new file in /var/log folder

4. Check filebeat log (/var/log/filebeat/filebeat) that the harvester started for new file

Logstash receiving data from Filebeat :-

Instead of sending Logstash output to ES please direct it to stdout.

Logstash.conf
------------------------

input {
  beats {
    port => 5044
  }
}

output {
	stdout { 
	     codec => rubydebug { } 
	}
}

[Vladpov]

After I added Logging part that you suggested into filebeat.yml I cannot even start filebeat. It gives me following message when I try to find out status of filebeat

● filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2019-10-25 07:06:57 UTC; 5s ago
 Docs: https://www.elastic.co/products/beats/filebeat
  Process: 2313 ExecStart=/usr/share/filebeat/bin/filebeat $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS (code=exited, status=1/FAILURE)
 Main PID: 2313 (code=exited, status=1/FAILURE)

Oct 25 07:06:57 bak2 systemd[1]: filebeat.service: Service hold-off time over, scheduling restart.
Oct 25 07:06:57 bak2 systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 5.
Oct 25 07:06:57 bak2 systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
Oct 25 07:06:57 bak2 systemd[1]: filebeat.service: Start request repeated too quickly.
Oct 25 07:06:57 bak2 systemd[1]: filebeat.service: Failed with result 'exit-code'.
Oct 25 07:06:57 bak2 systemd[1]: Failed to start Filebeat sends log files to Logstash or directly to Elasticsearch..

[kumarabhi (Abhishek)]

YAML files can be very sensitive to space and indentation.
Please check filebeat_reference.yml and copy the logging part into filebeat.yml
Make changes to settings if required and then try starting Filebeat again