Hi, I've got an ElasticStack set up as per these instructions: https://holdmybeersecurity.com/2019/05/01/back-in-the-saddle-install-setup-elastic-stack-7-0-on-ubuntu-18-04/
What I'm trying to do is get Zeek logs into the Elastic Stack. Kibana, Elasticsearch, Logstash, Filebeats and Zeek are all working. I can see Zeek's dns.log, ssl.log, dhcp.log, conn.log and everything else in Kibana except http.log. I'm not sure where the problem is and I'm hoping someone can help out. If I cat the http.log the data in the file is present and correct so Zeek is logging the data but it just won't come into Kibana.
I ran bin/filebeat test config --path.config /etc/filebeat and there's even the harvester for http.log:
2020-01-31T23:32:02.877Z INFO instance/beat.go:297 Setup Beat: filebeat; Version: 7.5.2
2020-01-31T23:32:02.877Z INFO [publisher] pipeline/module.go:97 Beat name: qelk
2020-01-31T23:32:02.877Z ERROR fileset/modules.go:125 Not loading modules. Module directory not found: /usr/share/filebeat/bin/module
2020-01-31T23:32:02.878Z WARN beater/filebeat.go:152 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-01-31T23:32:02.878Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-01-31T23:32:02.878Z INFO instance/beat.go:429 filebeat start running.
2020-01-31T23:32:02.878Z INFO registrar/migrate.go:104 No registry home found. Create: /usr/share/filebeat/bin/data/registry/filebeat
2020-01-31T23:32:02.960Z INFO registrar/migrate.go:112 Initialize registry meta file
2020-01-31T23:32:02.970Z INFO registrar/registrar.go:108 No registry file found under: /usr/share/filebeat/bin/data/registry/filebeat/data.json. Creating a new registry file.
2020-01-31T23:32:02.979Z INFO registrar/registrar.go:145 Loading registrar data from /usr/share/filebeat/bin/data/registry/filebeat/data.json
2020-01-31T23:32:02.980Z INFO registrar/registrar.go:152 States Loaded from registrar: 0
2020-01-31T23:32:02.980Z WARN beater/filebeat.go:368 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-01-31T23:32:02.980Z INFO crawler/crawler.go:72 Loading Inputs: 1
2020-01-31T23:32:02.980Z INFO log/input.go:152 Configured paths: [/opt/zeek/logs/current/*.log]
2020-01-31T23:32:02.980Z INFO input/input.go:114 Starting input of type: log; ID: 2622283721725469835
2020-01-31T23:32:02.981Z ERROR fileset/modules.go:125 Not loading modules. Module directory not found: /usr/share/filebeat/bin/module
2020-01-31T23:32:02.981Z INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2020-01-31T23:32:02.981Z INFO cfgfile/reload.go:171 Config reloader started
2020-01-31T23:32:02.981Z ERROR fileset/modules.go:125 Not loading modules. Module directory not found: /usr/share/filebeat/bin/module
2020-01-31T23:32:02.981Z INFO cfgfile/reload.go:226 Loading of config files completed.
2020-01-31T23:32:02.981Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/dns.log
2020-01-31T23:32:02.983Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/files.log
2020-01-31T23:32:02.986Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/dhcp.log
2020-01-31T23:32:02.986Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/notice.log
2020-01-31T23:32:02.986Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/software.log
2020-01-31T23:32:02.987Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/http.log
2020-01-31T23:32:02.987Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/ntp.log
2020-01-31T23:32:02.987Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/stdout.log
2020-01-31T23:32:02.987Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/weird.log
2020-01-31T23:32:02.987Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/capture_loss.log
2020-01-31T23:32:02.987Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/ssl.log
2020-01-31T23:32:02.988Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/known_hosts.log
2020-01-31T23:32:02.988Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/stats.log
2020-01-31T23:32:02.988Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/stderr.log
2020-01-31T23:32:02.988Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/x509.log
2020-01-31T23:32:02.988Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/conn.log
2020-01-31T23:32:02.989Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/known_services.log
2020-01-31T23:32:03.008Z INFO pipeline/output.go:95 Connecting to backoff(async(tcp://127.0.0.1:5044))
2020-01-31T23:32:03.008Z INFO pipeline/output.go:105 Connection to backoff(async(tcp://127.0.0.1:5044)) established
Here's my configs
filebeat.yml:
###################### Filebeat Configuration Example #########################
# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html
# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.
#=========================== Filebeat inputs =============================
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
# - /opt/bro/logs/current/*.log
- /opt/zeek/logs/current/*.log
# - /opt/zeek/logs/current/http.log
# Exclude lines. A list of regular expressions to match. It drops the lines that are
# matching any regular expression from the list.
#exclude_lines: ['^DBG']
# Include lines. A list of regular expressions to match. It exports the lines that are
# matching any regular expression from the list.
#include_lines: ['^ERR', '^WARN']
# Exclude files. A list of regular expressions to match. Filebeat drops the files that
# are matching any regular expression from the list. By default, no files are dropped.
#exclude_files: ['.gz$']
# Optional additional fields. These fields can be freely picked
# to add additional information to the crawled log files for filtering
#fields:
# level: debug
# review: 1
### Multiline options
# Multiline can be used for log messages spanning multiple lines. This is common
# for Java Stack Traces or C-Line Continuation
# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
#multiline.pattern: ^\[
# Defines if the pattern set under pattern should be negated or not. Default is false.
#multiline.negate: false
# Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
# that was (not) matched before or after or as long as a pattern is not matched based on negate.
# Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
#multiline.match: after
#============================= Filebeat modules ===============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
#================================ General =====================================
# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:
# The tags of the shipper are included in their own field with each
# transaction published.
tags: ["zeek"]
# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging
#================================ Outputs =====================================
# Configure what output to use when sending the data collected by the beat.
#----------------------------- Logstash output --------------------------------
output.logstash:
# The Logstash hosts
hosts: ["127.0.0.1:5044"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/filebeat/ssl/logstash.crt"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
#output.elasticsearch:
# hosts: ["127.0.0.1:9200"]
#setup.kibana:
# host: "127.0.0.1"
Logstash beats config file: 01-BeatsConfig.conf
########################################################################################
# Inputs are used to ingest logs from remote logging clients
########################################################################################
input {
# Ingest logs that match the Beat template
beats {
# Accept connections on port 5044
port => 5044
# Accept SSL connections
#ssl => true
# Public cert files
#ssl_certificate => "/etc/logstash/ssl/logstash.crt"
#ssl_key => "/etc/logstash/ssl/logstash.key"
# Do not verify client
#ssl_verify_mode => "none"
}
}
########################################################################################
# Filters are used to transform and modify the logs
########################################################################################
filter {
# Only apply these transformations to logs that contain the "Zeek" tag
if "zeek" in [tags] {
# Extract the json into Key value pairs
json {
source => "message"
}
# Remove the message field because it was extracted above
mutate {
remove_field => ["message"]
}
# Rename field names
mutate {
rename => ["id.orig_h", "src_ip" ]
rename => ["id.orig_p", "src_port" ]
rename => ["id.resp_h", "dst_ip" ]
rename => ["id.resp_p", "dst_port" ]
rename => ["host.name", "hostname" ]
}
}
}
########################################################################################
# Outputs take the logs and output them to a long term storage
########################################################################################
output {
# Send logs that contain the zeek tag too
if "zeek" in [tags] {
# Outputting logs to elasticsearch
elasticsearch {
# ES host to send logs too
hosts => ["http://localhost:9200"]
# Index to store data in
index => "zeek-%{+YYYY.MM.dd}"
}
}
}
Why can I get everything BUT http.log in Kibana?