# I can't see Zeek's http.log in Kibana but everything else (DNS, SSL, etc.) is fine

**URL:** <https://discuss.elastic.co/t/i-cant-see-zeeks-http-log-in-kibana-but-everything-else-dns-ssl-etc-is-fine/217474>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [January 31, 2020, 11:53pm UTC](https://discuss.elastic.co/t/i-cant-see-zeeks-http-log-in-kibana-but-everything-else-dns-ssl-etc-is-fine/217474 "2020-01-31T23:53:56Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sylancer](https://avatars.discourse-cdn.com/v4/letter/s/da6949/32.png) [@Sylancer](https://discuss.elastic.co/u/Sylancer)\
**Post date:** [January 31, 2020, 11:53pm UTC](https://discuss.elastic.co/t/i-cant-see-zeeks-http-log-in-kibana-but-everything-else-dns-ssl-etc-is-fine/217474/1 "2020-01-31T23:53:56Z")

</div>

Hi, I've got an ElasticStack set up as per these instructions: [https://holdmybeersecurity.com/2019/05/01/back-in-the-saddle-install-setup-elastic-stack-7-0-on-ubuntu-18-04/](https://holdmybeersecurity.com/2019/05/01/back-in-the-saddle-install-setup-elastic-stack-7-0-on-ubuntu-18-04/)

What I'm trying to do is get Zeek logs into the Elastic Stack. Kibana, Elasticsearch, Logstash, Filebeats and Zeek are all working. I can see Zeek's dns.log, ssl.log, dhcp.log, conn.log and everything else in Kibana except http.log. I'm not sure where the problem is and I'm hoping someone can help out. If I cat the http.log the data in the file is present and correct so Zeek is logging the data but it just won't come into Kibana.

I ran `bin/filebeat test config --path.config /etc/filebeat` and there's even the harvester for http.log:

```
2020-01-31T23:32:02.877Z INFO instance/beat.go:297 Setup Beat: filebeat; Version: 7.5.2
2020-01-31T23:32:02.877Z INFO [publisher] pipeline/module.go:97 Beat name: qelk
2020-01-31T23:32:02.877Z ERROR fileset/modules.go:125 Not loading modules. Module directory not found: /usr/share/filebeat/bin/module
2020-01-31T23:32:02.878Z WARN beater/filebeat.go:152 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-01-31T23:32:02.878Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-01-31T23:32:02.878Z INFO instance/beat.go:429 filebeat start running.
2020-01-31T23:32:02.878Z INFO registrar/migrate.go:104 No registry home found. Create: /usr/share/filebeat/bin/data/registry/filebeat
2020-01-31T23:32:02.960Z INFO registrar/migrate.go:112 Initialize registry meta file
2020-01-31T23:32:02.970Z INFO registrar/registrar.go:108 No registry file found under: /usr/share/filebeat/bin/data/registry/filebeat/data.json. Creating a new registry file.
2020-01-31T23:32:02.979Z INFO registrar/registrar.go:145 Loading registrar data from /usr/share/filebeat/bin/data/registry/filebeat/data.json
2020-01-31T23:32:02.980Z INFO registrar/registrar.go:152 States Loaded from registrar: 0
2020-01-31T23:32:02.980Z WARN beater/filebeat.go:368 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-01-31T23:32:02.980Z INFO crawler/crawler.go:72 Loading Inputs: 1
2020-01-31T23:32:02.980Z INFO log/input.go:152 Configured paths: [/opt/zeek/logs/current/*.log]
2020-01-31T23:32:02.980Z INFO input/input.go:114 Starting input of type: log; ID: 2622283721725469835
2020-01-31T23:32:02.981Z ERROR fileset/modules.go:125 Not loading modules. Module directory not found: /usr/share/filebeat/bin/module
2020-01-31T23:32:02.981Z INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2020-01-31T23:32:02.981Z INFO cfgfile/reload.go:171 Config reloader started
2020-01-31T23:32:02.981Z ERROR fileset/modules.go:125 Not loading modules. Module directory not found: /usr/share/filebeat/bin/module
2020-01-31T23:32:02.981Z INFO cfgfile/reload.go:226 Loading of config files completed.
2020-01-31T23:32:02.981Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/dns.log
2020-01-31T23:32:02.983Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/files.log
2020-01-31T23:32:02.986Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/dhcp.log
2020-01-31T23:32:02.986Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/notice.log
2020-01-31T23:32:02.986Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/software.log
2020-01-31T23:32:02.987Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/http.log
2020-01-31T23:32:02.987Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/ntp.log
2020-01-31T23:32:02.987Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/stdout.log
2020-01-31T23:32:02.987Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/weird.log
2020-01-31T23:32:02.987Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/capture_loss.log
2020-01-31T23:32:02.987Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/ssl.log
2020-01-31T23:32:02.988Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/known_hosts.log
2020-01-31T23:32:02.988Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/stats.log
2020-01-31T23:32:02.988Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/stderr.log
2020-01-31T23:32:02.988Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/x509.log
2020-01-31T23:32:02.988Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/conn.log
2020-01-31T23:32:02.989Z INFO log/harvester.go:251 Harvester started for file: /opt/zeek/logs/current/known_services.log
2020-01-31T23:32:03.008Z INFO pipeline/output.go:95 Connecting to backoff(async(tcp://127.0.0.1:5044))
2020-01-31T23:32:03.008Z INFO pipeline/output.go:105 Connection to backoff(async(tcp://127.0.0.1:5044)) established

```

Here's my configs

filebeat.yml:

```
###################### Filebeat Configuration Example #########################

# This file is an example configuration file highlighting only the most common
# options. The filebeat.reference.yml file from the same directory contains all the
# supported options with more comments. You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html

# For more available modules and options, please see the filebeat.reference.yml sample
# configuration file.

#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.

- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
# - /opt/bro/logs/current/*.log
    - /opt/zeek/logs/current/*.log
# - /opt/zeek/logs/current/http.log

  # Exclude lines. A list of regular expressions to match. It drops the lines that are
  # matching any regular expression from the list.
  #exclude_lines: ['^DBG']

  # Include lines. A list of regular expressions to match. It exports the lines that are
  # matching any regular expression from the list.
  #include_lines: ['^ERR', '^WARN']

  # Exclude files. A list of regular expressions to match. Filebeat drops the files that
  # are matching any regular expression from the list. By default, no files are dropped.
  #exclude_files: ['.gz$']

  # Optional additional fields. These fields can be freely picked
  # to add additional information to the crawled log files for filtering
  #fields:
  # level: debug
  # review: 1

  ### Multiline options

  # Multiline can be used for log messages spanning multiple lines. This is common
  # for Java Stack Traces or C-Line Continuation
  # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
  #multiline.pattern: ^\[

  # Defines if the pattern set under pattern should be negated or not. Default is false.
  #multiline.negate: false

  # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern
  # that was (not) matched before or after or as long as a pattern is not matched based on negate.
  # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash
  #multiline.match: after

#============================= Filebeat modules ===============================

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

#================================ General =====================================

# The name of the shipper that publishes the network data. It can be used to group
# all the transactions sent by a single shipper in the web interface.
#name:

# The tags of the shipper are included in their own field with each
# transaction published.
tags: ["zeek"]

# Optional fields that you can specify to add additional information to the
# output.
#fields:
# env: staging

#================================ Outputs =====================================
# Configure what output to use when sending the data collected by the beat.

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["127.0.0.1:5044"]

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/filebeat/ssl/logstash.crt"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"

#output.elasticsearch:
# hosts: ["127.0.0.1:9200"]
#setup.kibana:
# host: "127.0.0.1"

```

Logstash beats config file: 01-BeatsConfig.conf

```
    ########################################################################################
    # Inputs are used to ingest logs from remote logging clients
    ########################################################################################
    input {
      # Ingest logs that match the Beat template
      beats {
        # Accept connections on port 5044
        port => 5044

        # Accept SSL connections
        #ssl => true

        # Public cert files
        #ssl_certificate => "/etc/logstash/ssl/logstash.crt"
        #ssl_key => "/etc/logstash/ssl/logstash.key"

        # Do not verify client
        #ssl_verify_mode => "none"

      }
    }

    ########################################################################################
    # Filters are used to transform and modify the logs
    ########################################################################################
    filter {
        # Only apply these transformations to logs that contain the "Zeek" tag
        if "zeek" in [tags] {
            # Extract the json into Key value pairs
            json {
                source => "message"
            }

            # Remove the message field because it was extracted above
            mutate {
                remove_field => ["message"]
            }

            # Rename field names
            mutate {
                rename => ["id.orig_h", "src_ip"]
                rename => ["id.orig_p", "src_port"]
                rename => ["id.resp_h", "dst_ip"]
                rename => ["id.resp_p", "dst_port"]
                rename => ["host.name", "hostname"]
            }
        }
    }

########################################################################################
# Outputs take the logs and output them to a long term storage
########################################################################################
output {
  # Send logs that contain the zeek tag too
  if "zeek" in [tags] {
    # Outputting logs to elasticsearch
    elasticsearch {
      # ES host to send logs too
      hosts => ["http://localhost:9200"]

      # Index to store data in
      index => "zeek-%{+YYYY.MM.dd}"
    }
  }
}

```

Why can I get everything BUT http.log in Kibana?

---

<div class="post-metadata">

**Author:** ![Sylancer](https://avatars.discourse-cdn.com/v4/letter/s/da6949/32.png) [@Sylancer](https://discuss.elastic.co/u/Sylancer)\
**Post date:** [February 3, 2020, 3:16pm UTC](https://discuss.elastic.co/t/i-cant-see-zeeks-http-log-in-kibana-but-everything-else-dns-ssl-etc-is-fine/217474/2 "2020-02-03T15:16:49Z")

</div>

I figured this out. It was the logstash filter. It should have looked more like this

```
########################################################################################
# Filters are used to transform and modify the logs
########################################################################################
filter {
    if "zeek" in [tags] {
        mutate {
            rename => ["id.orig_h", "src_ip"]
            rename => ["id.orig_p", "src_port"]
            rename => ["id.resp_h", "dst_ip"]
            rename => ["id.resp_p", "dst_port"]
            rename => ["host.name", "hostname"]
        }
    }
}

```

Now I just have to figure out how to parse the http message field.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 2, 2020, 3:16pm UTC](https://discuss.elastic.co/t/i-cant-see-zeeks-http-log-in-kibana-but-everything-else-dns-ssl-etc-is-fine/217474/3 "2020-03-02T15:16:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.