Filebeat 6.2 and Windows

Diggy · February 9, 2018, 11:08pm

I recently upgraded my stack to 6.2 from 5.7.1, and struggled to get Filebeat working. On my Linux hosts, I had to upgrade to Filebeat-6.2, as well as to create a new Filebeat index based on Filebeat-6.2 in my Elastic host. So, Filebeat is working on my Linux hosts.

Not so on my Windows hosts. I tried to follow the same procedure - upgraded to 6.2. However, at best, I can start the Filebeat service, only to have it die shortly thereafter. I believe this has to do with my SSL certificate, or lack thereof. If I add the following to filebeat.yml, the service errors/won't start:

ssl.certificate_authorities: ["C:\Program Files\filebeat\ELK.crt"]

Here's the entire section:

output.logstash:
  # The Logstash hosts
  hosts: ["10.0.101.101:5044"]
  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
  #ssl.certificate_authorities: ["C:\Program Files\filebeat\ELK.crt"]
  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"
  # Client Certificate Key
  #ssl.key: "C:\Program Files\filebeat\ELK.key"

This was the same directive (without the "ssl.") that I had used with the earlier version that was installed (1.3.1, iirc, or wjatever version worked with the 5.x Elastic stack). As mentioned earlier, if I remove that directive, the Filebeat service starts, but dies soon thereafter.

Can anyone help me get Filebeat working on Windows?

Thanks.

andrewkroh · February 10, 2018, 1:03pm

Please share the log from Filebeat. This should indicate why it is stopping. And please share the complete configuration you are using.

Did you fully uninstall the old version you had installed? Make sure that you ran the uninstall script, then run the install script. This will ensure that the windows service has the correct flags for starting Filebeat.

Diggy · February 10, 2018, 3:37pm

I did fully uninstall the old version (ran PS uninstall script, then ran PS install script from version 6.2). Logging doesn't run if the Filebeat service doesn't start. Here are my configs:

Logstash input:

input {
  beats {
    port => 5044
    type => "log"
    tags => ['log']
    ssl => true
    ssl_certificate => "/etc/pki/tls/certs/ELK.crt"
    ssl_key => "/etc/pki/tls/private/ELK.key"
  }
}

and filebeat.yml (enabled parts):

#=========================== Filebeat prospectors =============================

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- type: log

  # Change to true to enable this prospector configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
#- /var/log/*.log
- e:\lm\log*.txt
- e:\lm\tclweb\htdocs\utilities\debug\logs\log*error
- e:\lm\tclweb\cache\logs\log.*
#- c:\programdata\elasticsearch\logs\*

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ['^DBG']
  exclude_lines: ["(iocp[0-9]+ timeout | iocp[0-9]+ handshake failed: | reset by peer)|(MessageData)|nosock bgerror|NCGI_TYPE_ENVACCESS|unknown protocol|chunk.field_index"]

#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#==================== Elasticsearch template setting ==========================

setup.template.settings:
  index.number_of_shards: 3
  #index.codec: best_compression
  #_source.enabled: false

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
#  env: staging


#============================== Dashboards =====================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here, or by using the `-setup` CLI flag or the `setup` command.
setup.dashboards.enabled: true

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "10.0.101.101:5601"

#================================ Outputs =====================================

# Configure what output to use when sending the data collected by the beat.

#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["10.0.101.101:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
  ssl.certificate_authorities: ["C:\Program Files\filebeat\ELK.crt"]
  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "C:\Program Files\filebeat\ELK.key"

#================================ Logging =====================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

#============================== Xpack Monitoring ===============================
# filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#xpack.monitoring.enabled: false

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well. Any setting that is not set is
# automatically inherited from the Elasticsearch output configuration, so if you
# have the Elasticsearch output configured, you can simply uncomment the
# following line.
#xpack.monitoring.elasticsearch:

Diggy · February 10, 2018, 4:36pm

The only way for me to get the filebeat service started is to set up filebeat.yml like this: https://pastebin.com/Jm5BZs7W . However, the logs don't seem to get to logstash. Here's some log output from the host running Filebeat: https://pastebin.com/hAf4KiHF

Diggy · February 12, 2018, 2:28pm

Partially solved - Filebeat was never going to work properly if I couldn't get the SSL part working. Trial and error led me to put escape characters in the path to the SSL cert. So, the directive:

ssl.certificate_authorities: ["C:\Program Files\filebeat\ELK.crt"]

became:

ssl.certificate_authorities: ["C:\\Program Files\\filebeat\\ELK.crt"]

So, now log data are being shipped. However, now I'm presented with another problem. The particular logs that I want to capture are retained for three months on their particular servers. Each file is timestamped, of course, as are the individual entries in the logs. New files are not created every day; data are appended to the logs until either a certain line count is reached, or the associated application is restarted.However, I only want to go back two weeks. If I add the ignore_older directive, and set the time period to be 336h (two weeks), I stop receiving logs. Once I comment out that directive, I receive thousands of hits per refresh interval. Is the Ignore_older directive no longer valid? How do I achieve what I'm trying to do?

andrewkroh · February 12, 2018, 7:28pm

ignore_older works based on the file modification time. Filebeat will not open files that have modification times older than the specified period. Can you check if the file modification dates are being updated?

Since Filebeat does not parse timestamps in log lines you could end up with more data being ingested than you expect due to the size based rotation strategy.

I'd recommend using a grok and date filter in Logstash to parse timestamps from log lines. Then apply a drop filter based on the timestamp.

Diggy · February 14, 2018, 2:07pm

I got everything working properly. The ignore_older directive works fine. As usual, I was just impatient; I guess filebeat had to cycle through the log files the first time. Anyway, I very much appreciate your input.

system · March 14, 2018, 2:07pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.