# Configure pfsense to ELK

**URL:** <https://discuss.elastic.co/t/configure-pfsense-to-elk/249360>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [September 21, 2020, 12:47pm UTC](https://discuss.elastic.co/t/configure-pfsense-to-elk/249360 "2020-09-21T12:47:03Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![Lusca\_lusca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lusca_lusca/32/45091_2.png) [@Lusca\_lusca](https://discuss.elastic.co/u/Lusca_lusca)\
**Post date:** [September 21, 2020, 12:47pm UTC](https://discuss.elastic.co/t/configure-pfsense-to-elk/249360/1 "2020-09-21T12:47:03Z")

</div>

Hi  
I am an intern at an IT company and I have to set up ELK to get logs from pfsense firewall, I am doing it all by myself but I don't have much knowledge about the topic. I am using an Azure server that I acess with my computer via ssh. In this server it is installed ubuntu 18.04 server that has no interface, thus I have been using only it's terminal. I installed elastic kibana logstash and filebeats. I can access kibana with my computer browser so I think I installed it correctly. The proble is that I am not receinving any logs from pfsense, at least I can't see it on kibana. I know that the logs are coming in because I use the command, nc -l -u 10.0.1.4 5002 so I can see the logs coming in. If you could tell me what do I have to show you guys so you can help me configure ELK I would appreciate it a lot. Thank you

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [September 21, 2020, 1:34pm UTC](https://discuss.elastic.co/t/configure-pfsense-to-elk/249360/3 "2020-09-21T13:34:19Z")

</div>

Could you please share your Beats configuration formatted using `</>` and its debug logs? `./filebeat -e -d "*"`?

---

<div class="post-metadata">

**Author:** ![Lusca\_lusca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lusca_lusca/32/45091_2.png) [@Lusca\_lusca](https://discuss.elastic.co/u/Lusca_lusca)\
**Post date:** [September 21, 2020, 1:52pm UTC](https://discuss.elastic.co/t/configure-pfsense-to-elk/249360/4 "2020-09-21T13:52:31Z")

</div>

```
input {
  beats {
    type => "pfsense"
    port => 5002
  } 
}
```

---

<div class="post-metadata">

**Author:** ![Lusca\_lusca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lusca_lusca/32/45091_2.png) [@Lusca\_lusca](https://discuss.elastic.co/u/Lusca_lusca)\
**Post date:** [September 21, 2020, 1:54pm UTC](https://discuss.elastic.co/t/configure-pfsense-to-elk/249360/5 "2020-09-21T13:54:16Z")

</div>

```
 filter {
  if "::" in [message] {
    grok {

      match => { "message" => "%{GREEDYDATA}"}
    }
  }
  else {
    grok {

      match => { "message" => "%{GREEDYDATA}"}
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![Lusca\_lusca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lusca_lusca/32/45091_2.png) [@Lusca\_lusca](https://discuss.elastic.co/u/Lusca_lusca)\
**Post date:** [September 21, 2020, 1:55pm UTC](https://discuss.elastic.co/t/configure-pfsense-to-elk/249360/6 "2020-09-21T13:55:55Z")

</div>

```
output {
        elasticsearch {
                hosts => ["http://localhost:9200"]
                index => "pf-%{+YYYY.MM.dd}" }
}
```

---

<div class="post-metadata">

**Author:** ![Lusca\_lusca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lusca_lusca/32/45091_2.png) [@Lusca\_lusca](https://discuss.elastic.co/u/Lusca_lusca)\
**Post date:** [September 21, 2020, 1:57pm UTC](https://discuss.elastic.co/t/configure-pfsense-to-elk/249360/7 "2020-09-21T13:57:59Z")

</div>

2020-09-21T13:57:28.766Z INFO instance/beat.go:640 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]  
2020-09-21T13:57:28.766Z DEBUG [beat] instance/beat.go:692 Beat metadata path: /var/lib/filebeat/meta.json  
2020-09-21T13:57:28.766Z INFO instance/beat.go:648 Beat ID: b5cc6468-fd1d-4123-8fc8-dc5bc9ffa8d5  
2020-09-21T13:57:28.766Z DEBUG [conditions] conditions/conditions.go:98 New condition contains: map  
2020-09-21T13:57:28.767Z DEBUG [conditions] conditions/conditions.go:98 New condition !contains: map  
2020-09-21T13:57:28.767Z DEBUG [add\_cloud\_metadata] add\_cloud\_metadata/providers.go:126 add\_cloud\_metadata: starting to fetch metadata, timeout=3s  
2020-09-21T13:57:28.767Z DEBUG [docker] docker/client.go:48 Docker client will negotiate the API version on the first request.  
2020-09-21T13:57:28.768Z DEBUG [add\_docker\_metadata] add\_docker\_metadata/add\_docker\_metadata.go:87 add\_docker\_metadata: docker environment not detected: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?  
2020-09-21T13:57:28.768Z DEBUG [kubernetes] add\_kubernetes\_metadata/kubernetes.go:138 Could not create kubernetes client using in\_cluster config: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES\_MASTER environment variable {"libbeat.processor": "add\_kubernetes\_metadata"}  
2020-09-21T13:57:28.773Z DEBUG [add\_cloud\_metadata] add\_cloud\_metadata/providers.go:162 add\_cloud\_metadata: received disposition for gcp after 6.395114ms. result=[provider:gcp, error=failed with http status code 404, metadata={}]  
2020-09-21T13:57:28.773Z DEBUG [add\_cloud\_metadata] add\_cloud\_metadata/providers.go:162 add\_cloud\_metadata: received disposition for openstack after 6.513214ms. result=[provider:openstack, error=failed with http status code 404, metadata={}]  
2020-09-21T13:57:28.775Z DEBUG [add\_cloud\_metadata] add\_cloud\_metadata/providers.go:162 add\_cloud\_metadata: received disposition for aws after 7.964917ms. result=[provider:aws, error=failed with http status code 404, metadata={}]  
2020-09-21T13:57:28.776Z DEBUG [add\_cloud\_metadata] add\_cloud\_metadata/providers.go:162 add\_cloud\_metadata: received disposition for azure after 9.536421ms. result=[provider:azure, error=, metadata={"instance":{"id":"2e5980f2-9971-4e04-a5c5-d60b2930151c","name":"Ubuntu-ELK"},"machine":{"type":"Standard\_B2s"},"provider":"azure","region":"eastus"}]  
2020-09-21T13:57:28.776Z DEBUG [add\_cloud\_metadata] add\_cloud\_metadata/providers.go:129 add\_cloud\_metadata: fetchMetadata ran for 9.587121ms  
2020-09-21T13:57:28.776Z INFO [add\_cloud\_metadata] add\_cloud\_metadata/add\_cloud\_metadata.go:93 add\_cloud\_metadata: hosting provider type detected as azure, metadata={"instance":{"id":"2e5980f2-9971-4e04-a5c5-d60b2930151c","name":"Ubuntu-ELK"},"machine":{"type":"Standard\_B2s"},"provider":"azure","region":"eastus"}  
2020-09-21T13:57:28.777Z DEBUG [processors] processors/processor.go:101 Generated new processors: add\_host\_metadata=[netinfo.enabled=[true], cache.ttl=[5m0s]], condition=!contains: map, add\_cloud\_metadata={"instance":{"id":"2e5980f2-9971-4e04-a5c5-d60b2930151c","name":"Ubuntu-ELK"},"machine":{"type":"Standard\_B2s"},"provider":"azure","region":"eastus"}, add\_docker\_metadata=[match\_fields= match\_pids=[process.pid, process.ppid]], add\_kubernetes\_metadata  
2020-09-21T13:57:28.777Z INFO instance/beat.go:387 filebeat stopped.  
2020-09-21T13:57:28.777Z ERROR instance/beat.go:951 Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).  
Exiting: data path already locked by another beat. Please make sure that multiple beats are not sharing the same data path (path.data).

---

<div class="post-metadata">

**Author:** ![Lusca\_lusca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lusca_lusca/32/45091_2.png) [@Lusca\_lusca](https://discuss.elastic.co/u/Lusca_lusca)\
**Post date:** [September 21, 2020, 2:05pm UTC](https://discuss.elastic.co/t/configure-pfsense-to-elk/249360/8 "2020-09-21T14:05:07Z")

</div>

I hope this is the configuration you are looking for

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [September 22, 2020, 2:05pm UTC](https://discuss.elastic.co/t/configure-pfsense-to-elk/249360/9 "2020-09-22T14:05:10Z")

</div>

I meant the Beats configuration.

Also, the debug logs of the Filebeat run you have shared are not informative. It looks like you already have a FB instance running in the background. Could you please stop that while you are testing?

---

<div class="post-metadata">

**Author:** ![Lusca\_lusca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lusca_lusca/32/45091_2.png) [@Lusca\_lusca](https://discuss.elastic.co/u/Lusca_lusca)\
**Post date:** [September 23, 2020, 1:36pm UTC](https://discuss.elastic.co/t/configure-pfsense-to-elk/249360/10 "2020-09-23T13:36:29Z")

</div>

```
###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.

# ============================== Filebeat inputs ===============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/*.log
    #- c:\programdata\elasticsearch\logs\*

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  # level: debug
  # review: 1

  ### Multiline options

  # Multiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation

  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  #multiline.pattern: ^\[

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  #multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  #multiline.match: after

# ============================== Filebeat modules ==============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

# ======================= Elasticsearch template setting =======================

setup.template.settings:
  index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false

# ================================== General ===================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging

# ================================= Dashboards =================================
# These settings control loading the sample dashboards to the Kibana index. Loading
# the dashboards is disabled by default and can be enabled either by setting the
# options here or by using the `setup` command.
setup.dashboards.enabled: true

# The URL from where to download the dashboards archive. By default this URL
# has a value which is computed based on the Beat name and version. For released
# versions, this URL points to the dashboard archive on the artifacts.elastic.co
# website.
#setup.dashboards.url:

# =================================== Kibana ===================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "localhost:5601"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

# =============================== Elastic Cloud ================================

# These settings simplify using Filebeat with the Elastic Cloud (https://cloud.elastic.co/).

# The cloud.id setting overwrites the `output.elasticsearch.hosts` and
# `setup.kibana.host` options.
# You can find the `cloud.id` in the Elastic Cloud web UI.
#cloud.id:

# The cloud.auth setting overwrites the `output.elasticsearch.username` and
# `output.elasticsearch.password` settings. The format is `<user>:<pass>`.
#cloud.auth:

# ================================== Outputs ===================================

# Configure what output to use when sending the data collected by the beat.

# ---------------------------- Elasticsearch Output ----------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
# hosts: ["localhost:9200"]

  # Protocol - either `http` (default) or `https`.
  #protocol: "https"

  # Authentication credentials - either API key or username/password.
  #api_key: "id:api_key"
  #username: "elastic"
  #password: "changeme"

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5002"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

# ================================= Processors =================================
processors:
  - add_host_metadata:
      when.not.contains.tags: forwarded
  - add_cloud_metadata: ~
  - add_docker_metadata: ~
  - add_kubernetes_metadata: ~

# ================================== Logging ===================================

# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]

# ============================= X-Pack Monitoring ==============================
# Filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster. This requires xpack monitoring to be enabled in Elasticsearch. The
# reporting is disabled by default.

# Set to true to enable the monitoring reporter.
#monitoring.enabled: false

# Sets the UUID of the Elasticsearch cluster under which monitoring data for this
# Filebeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
# is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
#monitoring.cluster_uuid:

# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well.
# Note that the settings should point to your Elasticsearch *monitoring* cluster.
# Any setting that is not set is automatically inherited from the Elasticsearch
# output configuration, so if you have the Elasticsearch output configured such
# that it is pointing to your Elasticsearch monitoring cluster, you can simply
# uncomment the following line.
#monitoring.elasticsearch:

# ============================== Instrumentation ===============================

# Instrumentation support for the filebeat.
#instrumentation:
    # Set to true to enable instrumentation of filebeat.
    #enabled: false

    # Environment in which filebeat is running on (eg: staging, production, etc.)
    #environment: ""

    # APM Server hosts to report instrumentation results to.
    #hosts:
    # - http://localhost:8200

    # API Key for the APM Server(s).
    # If api_key is set then secret_token will be ignored.
    #api_key:

    # Secret token for the APM Server(s).
    #secret_token:

# ================================= Migration ==================================

# This allows to enable 6.7 migration aliases
#migration.6_to_7.enabled: true
```

---

<div class="post-metadata">

**Author:** ![Lusca\_lusca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lusca_lusca/32/45091_2.png) [@Lusca\_lusca](https://discuss.elastic.co/u/Lusca_lusca)\
**Post date:** [September 23, 2020, 1:48pm UTC](https://discuss.elastic.co/t/configure-pfsense-to-elk/249360/11 "2020-09-23T13:48:47Z")

</div>

filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.  
Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)  
Active: active (running) since Fri 2020-09-18 12:55:58 UTC; 5 days ago  
Docs: [https://www.elastic.co/products/beats/filebeat](https://www.elastic.co/products/beats/filebeat)  
Main PID: 27760 (filebeat)  
Tasks: 9 (limit: 4680)  
CGroup: /system.slice/filebeat.service  
└─27760 /usr/share/filebeat/bin/filebeat --environment systemd -c /etc/filebeat/filebeat.yml --path.home /usr/share/filebeat --path.config /etc/filebeat --path.data /var/

Sep 23 13:40:30 Ubuntu-ELK filebeat[27760]: 2020-09-23T13:40:30.898Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitorin  
Sep 23 13:41:00 Ubuntu-ELK filebeat[27760]: 2020-09-23T13:41:00.899Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitorin  
Sep 23 13:41:30 Ubuntu-ELK filebeat[27760]: 2020-09-23T13:41:30.898Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitorin  
Sep 23 13:41:57 Ubuntu-ELK filebeat[27760]: 2020-09-23T13:41:57.973Z INFO log/harvester.go:324 File is inactive: /var/log/auth.log. Closing because close\_inacti  
Sep 23 13:42:00 Ubuntu-ELK filebeat[27760]: 2020-09-23T13:42:00.898Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitorin  
Sep 23 13:42:30 Ubuntu-ELK filebeat[27760]: 2020-09-23T13:42:30.898Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitorin  
Sep 23 13:43:00 Ubuntu-ELK filebeat[27760]: 2020-09-23T13:43:00.898Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitorin  
Sep 23 13:43:30 Ubuntu-ELK filebeat[27760]: 2020-09-23T13:43:30.898Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitorin  
Sep 23 13:44:00 Ubuntu-ELK filebeat[27760]: 2020-09-23T13:44:00.898Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitorin  
Sep 23 13:44:30 Ubuntu-ELK filebeat[27760]: 2020-09-23T13:44:30.898Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitorin

---

<div class="post-metadata">

**Author:** ![Lusca\_lusca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lusca_lusca/32/45091_2.png) [@Lusca\_lusca](https://discuss.elastic.co/u/Lusca_lusca)\
**Post date:** [September 23, 2020, 2:48pm UTC](https://discuss.elastic.co/t/configure-pfsense-to-elk/249360/12 "2020-09-23T14:48:32Z")

</div>

How do I know if there is another filebeat running on backgroud?

---

<div class="post-metadata">

**Author:** ![LeeDr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leedr/32/9289_2.png) [@LeeDr](https://discuss.elastic.co/u/LeeDr)\
**Post date:** [October 5, 2020, 6:29pm UTC](https://discuss.elastic.co/t/configure-pfsense-to-elk/249360/13 "2020-10-05T18:29:26Z")

</div>

`ps -ef | grep filebeat` should show it.  
If you find it and want to stop it you can `kill -9 <the pid number from that filebeat process>`

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 2, 2020, 8:29pm UTC](https://discuss.elastic.co/t/configure-pfsense-to-elk/249360/14 "2020-11-02T20:29:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.