Filebeat > Nginx Module

Hi all, if I perform a "filebeat export template > test.json" and the file showing there are fields that are required, e.g geo.location etc. But when I open the visualization, it prompted me "No Data". Why is it so ?

hi @skyluke.1987

filebeat export template > test.json

This command shows what template will be loaded when it is run not what IS currently loaded in the Elasticsearch cluster.

If you want to see what is currently loaded use this

curl http://localhost:9200/_template/filebeat-7.1.1

Or go to the Dev Tools and Run.

GET /_template/filebeat-7.1.1

Also confusing to me is that some of your screen shots show logstash are you using logstash as well? I would first get the simple Filebeat -> Elasticsearch directly.

The screen shot above shows the logstash output not filebeat setup that is a little more complex to setup.

Me if I were you I would start with a clean setup or you will need to remove the template, index patterns and existing filebeat indexes

Just to test it out...

I just built a brand new 7.1.1 single node Elasticsearch and Kibana on localhost and everything works fine the first time. I did not use Logstash.

I simply started Elasticsearch and Kibana without editing any settings.

Enabled ngnix module

./filebeat modules enable nginx

ran setup

./filebeat setup

Downloaded the nginx example logs file (see below for link).

edit the modules/nginx.yml and set the path to the nginx log file I just downloaded.

# Module: nginx
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.1/filebeat-module-nginx.html

- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: [ "/Users/sbrown/Downloads/nginx_logs.log" ]

Then started filebeat

./filebeat -e

The data loaded and the visualizations work fine with 1 exception there are no error logs so those are blank, this data set only contains access logs. The data is from May 2015.

Here is the data I loaded....

1 Like

Hi @stephenb, thanks for your detailed reply. Because of my setup involved these 2 different servers (A & B), but my output settings in Filebeat.yml is output to ES, is that how it works?

Secondly, I have the visualize template and the dashboards in the kibana, I am guessing that these has caused some issues when I re-run the ./filebeat setup command and also the load template command. May I know is there a method which I can clean this situation and do a clean install ?

image

Do you have this field / data in your index? When I open the dashboards, there are many info cannot be found.

Hi, for your reference these are the fields:

  "default_field" : [
    "message",
    "tags",
    "agent.ephemeral_id",
    "agent.id",
    "agent.name",
    "agent.type",
    "agent.version",
    "client.address",
    "client.domain",
    "client.geo.city_name",
    "client.geo.continent_name",
    "client.geo.country_iso_code",
    "client.geo.country_name",
    "client.geo.name",
    "client.geo.region_iso_code",
    "client.geo.region_name",
    "client.mac",
    "client.user.email",
    "client.user.full_name",
    "client.user.group.id",
    "client.user.group.name",
    "client.user.hash",
    "client.user.id",
    "client.user.name",
    "cloud.account.id",
    "cloud.availability_zone",
    "cloud.instance.id",
    "cloud.instance.name",
    "cloud.machine.type",
    "cloud.provider",
    "cloud.region",
    "container.id",
    "container.image.name",
    "container.image.tag",
    "container.name",
    "container.runtime",
    "destination.address",
    "destination.domain",
    "destination.geo.city_name",
    "destination.geo.continent_name",
    "destination.geo.country_iso_code",
    "destination.geo.country_name",
    "destination.geo.name",
    "destination.geo.region_iso_code",
    "destination.geo.region_name",
    "destination.mac",
    "destination.user.email",
    "destination.user.full_name",
    "destination.user.group.id",
    "destination.user.group.name",
    "destination.user.hash",
    "destination.user.id",
    "destination.user.name",
    "ecs.version",
    "error.code",
    "error.id",
    "error.message",
    "event.action",
    "event.category",
    "event.dataset",
    "event.hash",
    "event.id",
    "event.kind",
    "event.module",
    "event.original",
    "event.outcome",
    "event.timezone",
    "event.type",
    "file.device",
    "file.extension",
    "file.gid",
    "file.group",
    "file.inode",
    "file.mode",
    "file.owner",
    "file.path",
    "file.target_path",
    "file.type",
    "file.uid",
    "geo.city_name",
    "geo.continent_name",
    "geo.country_iso_code",
    "geo.country_name",
    "geo.name",
    "geo.region_iso_code",
    "geo.region_name",
    "group.id",
    "group.name",
    "host.architecture",
    "host.geo.city_name",
    "host.geo.continent_name",
    "host.geo.country_iso_code",
    "host.geo.country_name",
    "host.geo.name",
    "host.geo.region_iso_code",
    "host.geo.region_name",
    "host.hostname",
    "host.id",
    "host.mac",
    "host.name",
    "host.os.family",
    "host.os.full",
    "host.os.kernel",
    "host.os.name",
    "host.os.platform",
    "host.os.version",
    "host.type",
    "host.user.email",
    "host.user.full_name",

Part 2 :

    "host.user.group.id",
    "host.user.group.name",
    "host.user.hash",
    "host.user.id",
    "host.user.name",
    "http.request.body.content",
    "http.request.method",
    "http.request.referrer",
    "http.response.body.content",
    "http.version",
    "log.level",
    "log.original",
    "network.application",
    "network.community_id",
    "network.direction",
    "network.iana_number",
    "network.name",
    "network.protocol",
    "network.transport",
    "network.type",
    "observer.geo.city_name",
    "observer.geo.continent_name",
    "observer.geo.country_iso_code",
    "observer.geo.country_name",
    "observer.geo.name",
    "observer.geo.region_iso_code",
    "observer.geo.region_name",
    "observer.hostname",
    "observer.mac",
    "observer.os.family",
    "observer.os.full",
    "observer.os.kernel",
    "observer.os.name",
    "observer.os.platform",
    "observer.os.version",
    "observer.serial_number",
    "observer.type",
    "observer.vendor",
    "observer.version",
    "organization.id",
    "organization.name",
    "os.family",
    "os.full",
    "os.kernel",
    "os.name",
    "os.platform",
    "os.version",
    "process.args",
    "process.executable",
    "process.name",
    "process.title",
    "process.working_directory",
    "server.address",
    "server.domain",
    "server.geo.city_name",
    "server.geo.continent_name",
    "server.geo.country_iso_code",
    "server.geo.country_name",
    "server.geo.name",
    "server.geo.region_iso_code",
    "server.geo.region_name",
    "server.mac",
    "server.user.email",
    "server.user.full_name",
    "server.user.group.id",
    "server.user.group.name",
    "server.user.hash",
    "server.user.id",
    "server.user.name",
    "service.ephemeral_id",
    "service.id",
    "service.name",
    "service.state",
    "service.type",
    "service.version",
    "source.address",
    "source.domain",
    "source.geo.city_name",
    "source.geo.continent_name",
    "source.geo.country_iso_code",
    "source.geo.country_name",
    "source.geo.name",
    "source.geo.region_iso_code",
    "source.geo.region_name",
    "source.mac",
    "source.user.email",
    "source.user.full_name",
    "source.user.group.id",
    "source.user.group.name",
    "source.user.hash",
    "source.user.id",
    "source.user.name",
    "url.domain",
    "url.fragment",
    "url.full",
    "url.original",
    "url.password",
    "url.path",
    "url.query",
    "url.scheme",
    "url.username",
    "user.email",
    "user.full_name",
    "user.group.id",
    "user.group.name",
    "user.hash",
    "user.id",
    "user.name",
    "user_agent.device.name",
    "user_agent.name",
    "user_agent.original",
    "user_agent.os.family",
    "user_agent.os.full",
    "user_agent.os.kernel",
    "user_agent.os.name",
    "user_agent.os.platform",
    "user_agent.os.version",
    "user_agent.version",
    "agent.hostname",
    "error.type",
    "timeseries.instance",
    "cloud.project.id",
    "cloud.image.id",
    "host.os.build",
    "host.os.codename",
    "kubernetes.pod.name",
    "kubernetes.pod.uid",
    "kubernetes.namespace",
    "kubernetes.node.name",
    "kubernetes.replicaset.name",
    "kubernetes.deployment.name",
    "kubernetes.statefulset.name",
    "kubernetes.container.name",
    "kubernetes.container.image",
    "jolokia.agent.version",
    "jolokia.agent.id",
    "jolokia.server.product",
    "jolokia.server.version",
    "jolokia.server.vendor",
    "jolokia.url",
    "log.file.path",
    "log.source.address",
    "stream",
    "input.type",
    "syslog.severity_label",
    "syslog.facility_label",
    "process.program",
    "log.flags",
    "user_agent.os.full_name",
    "fileset.name",
    "event.code",
    "icmp.code",
    "icmp.type",
    "igmp.type",
    "source.as.organization.name",
    "destination.as.organization.name",
    "apache.access.ssl.protocol",
    "apache.access.ssl.cipher",
    "apache.error.module",
    "user.terminal",
    "user.audit.id",
    "user.audit.name",
    "user.audit.group.id",
    "user.audit.group.name",

Part 3:

    "user.effective.id",
    "user.effective.name",
    "user.effective.group.id",
    "user.effective.group.name",
    "user.filesystem.id",
    "user.filesystem.name",
    "user.filesystem.group.id",
    "user.filesystem.group.name",
    "user.owner.id",
    "user.owner.name",
    "user.owner.group.id",
    "user.owner.group.name",
    "user.saved.id",
    "user.saved.name",
    "user.saved.group.id",
    "user.saved.group.name",
    "auditd.log.old_auid",
    "auditd.log.new_auid",
    "auditd.log.old_ses",
    "auditd.log.new_ses",
    "auditd.log.items",
    "auditd.log.item",
    "auditd.log.tty",
    "auditd.log.a0",
    "cisco.asa.message_id",
    "cisco.asa.suffix",
    "cisco.asa.source_interface",
    "cisco.asa.destination_interface",
    "cisco.asa.list_id",
    "cisco.asa.source_username",
    "cisco.asa.destination_username",
    "cisco.asa.threat_level",
    "cisco.asa.threat_category",
    "cisco.asa.connection_id",
    "cisco.ios.access_list",
    "cisco.ios.facility",
    "coredns.id",
    "coredns.query.class",
    "coredns.query.name",
    "coredns.query.type",
    "coredns.response.code",
    "coredns.response.flags",
    "elasticsearch.component",
    "elasticsearch.cluster.uuid",
    "elasticsearch.cluster.name",
    "elasticsearch.node.id",
    "elasticsearch.node.name",
    "elasticsearch.index.name",
    "elasticsearch.index.id",
    "elasticsearch.shard.id",
    "elasticsearch.audit.layer",
    "elasticsearch.audit.event_type",
    "elasticsearch.audit.origin.type",
    "elasticsearch.audit.realm",
    "elasticsearch.audit.user.realm",
    "elasticsearch.audit.user.roles",
    "elasticsearch.audit.action",
    "elasticsearch.audit.url.params",
    "elasticsearch.audit.indices",
    "elasticsearch.audit.request.id",
    "elasticsearch.audit.request.name",
    "elasticsearch.audit.message",
    "elasticsearch.gc.phase.name",
    "elasticsearch.gc.tags",
    "elasticsearch.slowlog.logger",
    "elasticsearch.slowlog.took",
    "elasticsearch.slowlog.types",
    "elasticsearch.slowlog.stats",
    "elasticsearch.slowlog.search_type",
    "elasticsearch.slowlog.source_query",
    "elasticsearch.slowlog.extra_source",
    "elasticsearch.slowlog.total_hits",
    "elasticsearch.slowlog.total_shards",
    "elasticsearch.slowlog.routing",
    "elasticsearch.slowlog.id",
    "elasticsearch.slowlog.type",
    "envoyproxy.log_type",
    "envoyproxy.response_flags",
    "envoyproxy.request_id",
    "envoyproxy.authority",
    "envoyproxy.proxy_type",
    "googlecloud.vpcflow.reporter",
    "googlecloud.vpcflow.destination.instance.project_id",
    "googlecloud.vpcflow.destination.instance.region",
    "googlecloud.vpcflow.destination.instance.zone",
    "googlecloud.vpcflow.destination.vpc.project_id",
    "googlecloud.vpcflow.destination.vpc.vpc_name",
    "googlecloud.vpcflow.destination.vpc.subnetwork_name",
    "googlecloud.vpcflow.source.instance.project_id",
    "googlecloud.vpcflow.source.instance.region",
    "googlecloud.vpcflow.source.instance.zone",
    "googlecloud.vpcflow.source.vpc.project_id",
    "googlecloud.vpcflow.source.vpc.vpc_name",
    "googlecloud.vpcflow.source.vpc.subnetwork_name",
    "haproxy.frontend_name",
    "haproxy.backend_name",
    "haproxy.server_name",
    "haproxy.bind_name",
    "haproxy.error_message",
    "haproxy.source",
    "haproxy.termination_state",
    "haproxy.mode",
    "haproxy.http.response.captured_cookie",
    "haproxy.http.response.captured_headers",
    "haproxy.http.request.captured_cookie",
    "haproxy.http.request.captured_headers",
    "haproxy.http.request.raw_request_line",
    "icinga.debug.facility",
    "icinga.main.facility",
    "icinga.startup.facility",
    "iis.access.site_name",
    "iis.access.server_name",
    "iis.access.cookie",
    "iis.error.reason_phrase",
    "iis.error.queue_name",
    "iptables.fragment_flags",
    "iptables.input_device",
    "iptables.output_device",
    "iptables.tcp.flags",
    "iptables.ubiquiti.input_zone",
    "iptables.ubiquiti.output_zone",
    "iptables.ubiquiti.rule_number",
    "iptables.ubiquiti.rule_set",
    "kafka.log.component",
    "kafka.log.class",
    "kafka.log.trace.class",
    "kafka.log.trace.message",
    "kibana.log.tags",
    "kibana.log.state",
    "logstash.log.module",
    "text",
    "logstash.log.thread",
    "logstash.slowlog.module",
    "text",
    "logstash.slowlog.thread",
    "text",
    "logstash.slowlog.event",
    "logstash.slowlog.plugin_name",
    "logstash.slowlog.plugin_type",
    "text",
    "logstash.slowlog.plugin_params",
    "mongodb.log.component",
    "mongodb.log.context",
    "mssql.log.origin",
    "mysql.slowlog.query",
    "mysql.slowlog.schema",
    "mysql.slowlog.current_user",
    "mysql.slowlog.last_errno",
    "mysql.slowlog.killed",
    "mysql.slowlog.log_slow_rate_type",
    "mysql.slowlog.log_slow_rate_limit",
    "mysql.slowlog.innodb.trx_id",
    "nats.log.msg.type",
    "nats.log.msg.subject",
    "nats.log.msg.reply_to",
    "nats.log.msg.error.message",
    "nats.log.msg.queue_group",
    "osquery.result.name",
    "osquery.result.action",
    "osquery.result.host_identifier",
    "osquery.result.calendar_time",
    "panw.panos.ruleset",
    "panw.panos.source.zone",
    "panw.panos.source.interface",

Part 4: Final

    "panw.panos.destination.zone",
    "panw.panos.destination.interface",
    "panw.panos.network.pcap_id",
    "panw.panos.network.nat.community_id",
    "panw.panos.file.hash",
    "panw.panos.url.category",
    "panw.panos.flow_id",
    "panw.panos.threat.resource",
    "panw.panos.threat.id",
    "panw.panos.threat.name",
    "postgresql.log.timestamp",
    "postgresql.log.database",
    "postgresql.log.query",
    "rabbitmq.log.pid",
    "redis.log.role",
    "redis.slowlog.cmd",
    "redis.slowlog.key",
    "redis.slowlog.args",
    "santa.action",
    "santa.decision",
    "santa.reason",
    "santa.mode",
    "santa.disk.volume",
    "santa.disk.bus",
    "santa.disk.serial",
    "santa.disk.bsdname",
    "santa.disk.model",
    "santa.disk.fs",
    "santa.disk.mount",
    "certificate.common_name",
    "certificate.sha256",
    "hash.sha256",
    "suricata.eve.event_type",
    "suricata.eve.app_proto_orig",
    "suricata.eve.tcp.tcp_flags",
    "suricata.eve.tcp.tcp_flags_tc",
    "suricata.eve.tcp.state",
    "suricata.eve.tcp.tcp_flags_ts",
    "suricata.eve.fileinfo.sha1",
    "suricata.eve.fileinfo.state",
    "suricata.eve.fileinfo.sha256",
    "suricata.eve.fileinfo.md5",
    "suricata.eve.dns.type",
    "suricata.eve.dns.rrtype",
    "suricata.eve.dns.rrname",
    "suricata.eve.dns.rdata",
    "suricata.eve.dns.rcode",
    "suricata.eve.flow_id",
    "suricata.eve.email.status",
    "suricata.eve.http.redirect",
    "suricata.eve.http.protocol",
    "suricata.eve.http.http_content_type",
    "suricata.eve.in_iface",
    "suricata.eve.alert.category",
    "suricata.eve.alert.signature",
    "suricata.eve.ssh.client.proto_version",
    "suricata.eve.ssh.client.software_version",
    "suricata.eve.ssh.server.proto_version",
    "suricata.eve.ssh.server.software_version",
    "suricata.eve.tls.issuerdn",
    "suricata.eve.tls.sni",
    "suricata.eve.tls.version",
    "suricata.eve.tls.fingerprint",
    "suricata.eve.tls.serial",
    "suricata.eve.tls.subject",
    "suricata.eve.app_proto_ts",
    "suricata.eve.flow.state",
    "suricata.eve.flow.reason",
    "suricata.eve.app_proto_tc",
    "suricata.eve.smtp.rcpt_to",
    "suricata.eve.smtp.mail_from",
    "suricata.eve.smtp.helo",
    "suricata.eve.app_proto_expected",
    "system.auth.ssh.method",
    "system.auth.ssh.signature",
    "system.auth.ssh.event",
    "system.auth.sudo.error",
    "system.auth.sudo.tty",
    "system.auth.sudo.pwd",
    "system.auth.sudo.user",
    "system.auth.sudo.command",
    "system.auth.useradd.home",
    "system.auth.useradd.shell",
    "traefik.access.user_identifier",
    "traefik.access.frontend_name",
    "traefik.access.backend_url",
    "zeek.session_id",
    "zeek.connection.state",
    "zeek.connection.history",
    "zeek.connection.orig_l2_addr",
    "zeek.connection.resp_l2_addr",
    "zeek.dns.trans_id",
    "zeek.dns.query",
    "zeek.dns.qclass_name",
    "zeek.dns.qtype_name",
    "zeek.dns.rcode_name",
    "zeek.dns.answers",
    "zeek.http.status_msg",
    "zeek.http.info_msg",
    "zeek.http.tags",
    "zeek.http.password",
    "zeek.http.proxied",
    "zeek.http.client_header_names",
    "zeek.http.server_header_names",
    "zeek.http.orig_fuids",
    "zeek.http.orig_mime_types",
    "zeek.http.orig_filenames",
    "zeek.http.resp_fuids",
    "zeek.http.resp_mime_types",
    "zeek.http.resp_filenames",
    "zeek.files.fuid",
    "zeek.files.session_ids",
    "zeek.files.source",
    "zeek.files.analyzers",
    "zeek.files.mime_type",
    "zeek.files.filename",
    "zeek.files.parent_fuid",
    "zeek.files.md5",
    "zeek.files.sha1",
    "zeek.files.sha256",
    "zeek.files.extracted",
    "zeek.ssl.version",
    "zeek.ssl.cipher",
    "zeek.ssl.curve",
    "zeek.ssl.server_name",
    "zeek.ssl.next_protocol",
    "zeek.ssl.cert_chain",
    "zeek.ssl.cert_chain_fuids",
    "zeek.ssl.client_cert_chain",
    "zeek.ssl.client_cert_chain_fuids",
    "zeek.ssl.issuer",
    "zeek.ssl.client_issuer",
    "zeek.ssl.validation_status",
    "zeek.ssl.validation_code",
    "zeek.ssl.subject",
    "zeek.ssl.client_subject",
    "zeek.ssl.last_alert",
    "zeek.notice.connection_id",
    "zeek.notice.icmp_id",
    "zeek.notice.file.id",
    "zeek.notice.file.parent_id",
    "zeek.notice.file.source",
    "zeek.notice.file.mime_type",
    "zeek.notice.fuid",
    "zeek.notice.note",
    "zeek.notice.msg",
    "zeek.notice.sub",
    "zeek.notice.peer_name",
    "zeek.notice.peer_descr",
    "zeek.notice.actions",
    "zeek.notice.email_body_sections",
    "zeek.notice.email_delay_tokens",
    "zeek.notice.identifier",
    "fields.*"

Did that come from?

GET /_template/filebeat-7.1.1

If so further down you will see this.......which is correct. The out put is very long.

   "source" : {
      "properties" : {
        "geo" : {
          "properties" : {
            "region_iso_code" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "continent_name" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "city_name" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "country_iso_code" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "country_name" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "name" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "region_name" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "location" : {
              "type" : "geo_point"
            }
          }
        },

In the index pattern you should see these fields like this shown below...

if not delete that index pattern from the Kibana GUI
Mangement -> Index Patterns
and run ./filebeat setup again.

Apologies but I don't think I am able to help much more ... to do a clean install ... uninstall and re-install elasticsearch and make sure the the data directory under the elasticsearch install is removed before you reinstall.

Hi may I check with you. My client side is installing filebeat 7.3.0 while the ES server's Filebeat is on 7.1.1. Will there be an issue in terms of compatibility and the data format?

Hi @stephenb, I manage to get this fields, but the dashboard still cannot capture.

What I did is I removed the client Filebeat and installed the 7.1.1 version.

image

After that I notice I am facing some difficulties installing the logstash on the client server, its Centos 6.10 and with Java 1.8.0. Will there be any problem connecting these 2 servers ?

Hi is there anyone can help to troubleshoot this issue? thx

@stephenb , may I know what is your index name for this Nginx elastic ?

I came across some forum suggest that the template only recognize nginx-* indexes in order to load those templates.

If you use the filebeat nginx module with all the default settings the nginx logs will be indexed into indexes name with pattern filebeat-*

Hi bro, so this is expected? Are you able to help me on this issue? I have no solution to it

I found this online, wanted to load this template, but it fails. https://github.com/elastic/beats/blob/master/filebeat/module/nginx/_meta/kibana/7/dashboard/Filebeat-nginx-logs.json

Can anyone share why? is it incompatible ?

How can I check the Filebeat's dashboard compatibility and which version did I installed ?

Dear all, is there anyone can help on my question? It's been sometimes.

Almost 99% of my dashboards cannot display the data collected, while there are data collected from the other servers stored and received into our Elasticsearch DB. Wonder why the default dashboard cannot display?

I have tried various method to resolve this, but all of it just doesn't work.

Next, the indexes ./filebeat has been deleted and reindex previously, but yet there is no data is reflecting on the dashboards.

Hi, is there any command I can used to check what went wrong?