Hi
I am an intern at an IT company and I have to set up ELK to get logs from pfsense firewall, I am doing it all by myself but I don't have much knowledge about the topic. I am using an Azure server that I acess with my computer via ssh. In this server it is installed ubuntu 18.04 server that has no interface, thus I have been using only it's terminal. I installed elastic kibana logstash and filebeats. I can access kibana with my computer browser so I think I installed it correctly. The proble is that I am not receinving any logs from pfsense, at least I can't see it on kibana. I know that the logs are coming in because I use the command, nc -l -u 10.0.1.4 5002 so I can see the logs coming in. If you could tell me what do I have to show you guys so you can help me configure ELK I would appreciate it a lot. Thank you
Could you please share your Beats configuration formatted using </> and its debug logs? ./filebeat -e -d "*"?
input {
beats {
type => "pfsense"
port => 5002
}
} filter {
if "::" in [message] {
grok {
match => { "message" => "%{GREEDYDATA}"}
}
}
else {
grok {
match => { "message" => "%{GREEDYDATA}"}
}
}
}output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "pf-%{+YYYY.MM.dd}" }
}2020-09-21T13:57:28.766Z INFO instance/beat.go:640 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2020-09-21T13:57:28.766Z DEBUG [beat] instance/beat.go:692 Beat metadata path: /var/lib/filebeat/meta.json
2020-09-21T13:57:28.766Z INFO instance/beat.go:648 Beat ID: b5cc6468-fd1d-4123-8fc8-dc5bc9ffa8d5
2020-09-21T13:57:28.766Z DEBUG [conditions] conditions/conditions.go:98 New condition contains: map
2020-09-21T13:57:28.767Z DEBUG [conditions] conditions/conditions.go:98 New condition !contains: map
2020-09-21T13:57:28.767Z DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:126 add_cloud_metadata: starting to fetch metadata, timeout=3s
2020-09-21T13:57:28.767Z DEBUG [docker] docker/client.go:48 Docker client will negotiate the API version on the first request.
2020-09-21T13:57:28.768Z DEBUG [add_docker_metadata] add_docker_metadata/add_docker_metadata.go:87 add_docker_metadata: docker environment not detected: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
2020-09-21T13:57:28.768Z DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:138 Could not create kubernetes client using in_cluster config: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable {"libbeat.processor": "add_kubernetes_metadata"}
2020-09-21T13:57:28.773Z DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for gcp after 6.395114ms. result=[provider:gcp, error=failed with http status code 404, metadata={}]
2020-09-21T13:57:28.773Z DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for openstack after 6.513214ms. result=[provider:openstack, error=failed with http status code 404, metadata={}]
2020-09-21T13:57:28.775Z DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for aws after 7.964917ms. result=[provider:aws, error=failed with http status code 404, metadata={}]
2020-09-21T13:57:28.776Z DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for azure after 9.536421ms. result=[provider:azure, error=, metadata={"instance":{"id":"2e5980f2-9971-4e04-a5c5-d60b2930151c","name":"Ubuntu-ELK"},"machine":{"type":"Standard_B2s"},"provider":"azure","region":"eastus"}]
2020-09-21T13:57:28.776Z DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:129 add_cloud_metadata: fetchMetadata ran for 9.587121ms
2020-09-21T13:57:28.776Z INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:93 add_cloud_metadata: hosting provider type detected as azure, metadata={"instance":{"id":"2e5980f2-9971-4e04-a5c5-d60b2930151c","name":"Ubuntu-ELK"},"machine":{"type":"Standard_B2s"},"provider":"azure","region":"eastus"}
2020-09-21T13:57:28.777Z DEBUG [processors] processors/processor.go:101 Generated new processors: add_host_metadata=[netinfo.enabled=[true], cache.ttl=[5m0s]], condition=!contains: map, add_cloud_metadata={"instance":{"id":"2e5980f2-9971-4e04-a5c5-d60b2930151c","name":"Ubuntu-ELK"},"machine":{"type":"Standard_B2s"},"provider":"azure","region":"eastus"}, add_docker_metadata=[match_fields= match_pids=[process.pid, process.ppid]], add_kubernetes_metadata
2020-09-21T13:57:28.777Z INFO instance/beat.go:387 filebeat stopped.
2020-09-21T13:57:28.777Z ERROR instance/beat.go:951 Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).
Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).
I hope this is the configuration you are looking for
I meant the Beats configuration.
Also, the debug logs of the Filebeat run you have shared are not informative. It looks like you already have a FB instance running in the background. Could you please stop that while you are testing?
###################### Filebeat Configuration Example #########################
# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html
# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.
# ============================== Filebeat inputs ===============================
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /var/log/*.log
#- c:\programdata\elasticsearch\logs\*
# Exclude lines. A list of regular expressions to match. It drops the lines that are
# matching any regular expression from the list.
#exclude_lines: ['^DBG']
# Include lines. A list of regular expressions to match. It exports the lines that are
# matching any regular expression from the list.
#include_lines: ['^ERR', '^WARN']
# Exclude files. A list of regular expressions to match. Filebeat drops the files that
# are matching any regular expression from the list. By default, no files are dropped.
#exclude_files: ['.gz$']
# Optional additional fields. These fields can be freely picked
# to add additional information to the crawled log files for filtering
#fields:
# level: debug
# review: 1
### Multiline options
# Multiline can be used for log messages spanning multiple lines. This is common
# for Java Stack Traces or C-Line Continuation
# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
#multiline.pattern: ^\[
# Defines if the pattern set under pattern should be negated or not. Default is false.
#multiline.negate: false
# Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
# that was (not) matched before or after or as long as a pattern is not matched based on negate.
# Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
#multiline.match: after
# ============================== Filebeat modules ==============================
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
# ======================= Elasticsearch template setting =======================
setup.template.settings:
index.number_of_shards: 1
#index.codec: best_compression
#_source.enabled: false
# ================================== General ===================================
# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:
# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]
# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging
# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
setup.dashboards.enabled: true
# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:
# =================================== Kibana ===================================
# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:
# Kibana Host
# Scheme and port can be left out and will be set to the default (http and 5601)
# In case you specify and additional path, the scheme is required: http://localhost:5601/path
# IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
host: "localhost:5601"
# Kibana Space ID
# ID of the Kibana Space into which the dashboards should be loaded. By default,
# the Default Space will be used.
#space.id:
# =============================== Elastic Cloud ================================
# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).
# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:
# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:
# ================================== Outputs ===================================
# Configure what output to use when sending the data collected by the beat.
# ---------------------------- Elasticsearch Output ----------------------------
#output.elasticsearch:
# Array of hosts to connect to.
# hosts: ["localhost:9200"]
# Protocol - either `http` (default) or `https`.
#protocol: "https"
# Authentication credentials - either API key or username/password.
#api_key: "id:api_key"
#username: "elastic"
#password: "changeme"
# ------------------------------ Logstash Output -------------------------------
output.logstash:
# The Logstash hosts
hosts: ["localhost:5002"]
# Optional SSL. By default is off.
# List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
# Certificate for SSL client authentication
#ssl.certificate: "/etc/pki/client/cert.pem"
# Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"
# ================================= Processors =================================
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
# ================================== Logging ===================================
# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]
# ============================= X-Pack Monitoring ==============================
# Filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster. This requires xpack monitoring to be enabled in Elasticsearch. The
# reporting is disabled by default.
# Set to true to enable the monitoring reporter.
#monitoring.enabled: false
# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:
# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:
# ============================== Instrumentation ===============================
# Instrumentation support for the filebeat.
#instrumentation:
# Set to true to enable instrumentation of filebeat.
#enabled: false
# Environment in which filebeat is running on (eg: staging, production, etc.)
#environment: ""
# APM Server hosts to report instrumentation results to.
#hosts:
# - http://localhost:8200
# API Key for the APM Server(s).
# If api_key is set then secret_token will be ignored.
#api_key:
# Secret token for the APM Server(s).
#secret_token:
# ================================= Migration ==================================
# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: truefilebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2020-09-18 12:55:58 UTC; 5 days ago
Docs: https://www.elastic.co/products/beats/filebeat
Main PID: 27760 (filebeat)
Tasks: 9 (limit: 4680)
CGroup: /system.slice/filebeat.service
└─27760 /usr/share/filebeat/bin/filebeat --environment systemd -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/
Sep 23 13:40:30 Ubuntu-ELK filebeat[27760]: 2020-09-23T13:40:30.898Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitorin
Sep 23 13:41:00 Ubuntu-ELK filebeat[27760]: 2020-09-23T13:41:00.899Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitorin
Sep 23 13:41:30 Ubuntu-ELK filebeat[27760]: 2020-09-23T13:41:30.898Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitorin
Sep 23 13:41:57 Ubuntu-ELK filebeat[27760]: 2020-09-23T13:41:57.973Z INFO log/harvester.go:324 File is inactive: /var/log/auth.log. Closing because close_inacti
Sep 23 13:42:00 Ubuntu-ELK filebeat[27760]: 2020-09-23T13:42:00.898Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitorin
Sep 23 13:42:30 Ubuntu-ELK filebeat[27760]: 2020-09-23T13:42:30.898Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitorin
Sep 23 13:43:00 Ubuntu-ELK filebeat[27760]: 2020-09-23T13:43:00.898Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitorin
Sep 23 13:43:30 Ubuntu-ELK filebeat[27760]: 2020-09-23T13:43:30.898Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitorin
Sep 23 13:44:00 Ubuntu-ELK filebeat[27760]: 2020-09-23T13:44:00.898Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitorin
Sep 23 13:44:30 Ubuntu-ELK filebeat[27760]: 2020-09-23T13:44:30.898Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitorin
How do I know if there is another filebeat running on backgroud?
ps -ef | grep filebeat should show it.
If you find it and want to stop it you can kill -9 <the pid number from that filebeat process>
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.