Filebeat > Nginx Module

Hi MarianaD, what if I created the index pattern name this way filebeat-7.3.0* ? Will this affects the result?

Hi @stephenb, I have this issue when I open the visualization:
How do I fix this?

I notice I was running Filebeat 7.1 instead of the new Filebeat7.3.. How can I choose to run the new one ? I remembered i have installed the new one before.

Hi there, is there anyone encounter this problem and have solution to it?

  1. Can I know how can I print out the data / fields received, does it contain the required fields, e.g. geoip.

  2. How can I control the data fields, and what if there is no field captured, what should I do? Currently the visualization showing empty. Error: "Could not locate that index-pattern-field (id: source.address)" and more.

@skyluke.1987

Apologies but It's a bit difficult to know what state your whole system is in but if you're all on 7.3 then you could try to delete the index pattern in kibana and then run filebeat setup again with the nginx module enabled and it should create the correct index pattern.

However if you have data from old filebeats and new filebeats they may not all work with the new index pattern.

I started from scratch installed filebeat and enabled nginx module ran setup and then send directly to elasticsearch all the visualizations and dashboards load. That's not to say that you're not running into issues but from a clean configuration it should work

Hi @stephenb, thanks for your reply and analysis. Currently I am using version 7.1 with Kibana and Elasticsearch. I notice that the visualizations cannot find the correct indices and this has caused the template cannot be loaded. May I know is there a way where we can list out all the fields and from there re-link all the required fields.

Secondly, I notice that on the other end (The Nginx server) the log files (.access and .error) log files contain very little information. Things like locations all these are not available. This could be one of the reason why my Map's visualization cannot be loaded with "No Data Found".

Kindly advise and share your thoughts.

Hi all, if I perform a "filebeat export template > test.json" and the file showing there are fields that are required, e.g geo.location etc. But when I open the visualization, it prompted me "No Data". Why is it so ?

hi @skyluke.1987

filebeat export template > test.json

This command shows what template will be loaded when it is run not what IS currently loaded in the Elasticsearch cluster.

If you want to see what is currently loaded use this

curl http://localhost:9200/_template/filebeat-7.1.1

Or go to the Dev Tools and Run.

GET /_template/filebeat-7.1.1

Also confusing to me is that some of your screen shots show logstash are you using logstash as well? I would first get the simple Filebeat -> Elasticsearch directly.

The screen shot above shows the logstash output not filebeat setup that is a little more complex to setup.

Me if I were you I would start with a clean setup or you will need to remove the template, index patterns and existing filebeat indexes

Just to test it out...

I just built a brand new 7.1.1 single node Elasticsearch and Kibana on localhost and everything works fine the first time. I did not use Logstash.

I simply started Elasticsearch and Kibana without editing any settings.

Enabled ngnix module

./filebeat modules enable nginx

ran setup

./filebeat setup

Downloaded the nginx example logs file (see below for link).

edit the modules/nginx.yml and set the path to the nginx log file I just downloaded.

# Module: nginx
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.1/filebeat-module-nginx.html

- module: nginx
  # Access logs
  access:
    enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: [ "/Users/sbrown/Downloads/nginx_logs.log" ]

Then started filebeat

./filebeat -e

The data loaded and the visualizations work fine with 1 exception there are no error logs so those are blank, this data set only contains access logs. The data is from May 2015.

Here is the data I loaded....

1 Like

Hi @stephenb, thanks for your detailed reply. Because of my setup involved these 2 different servers (A & B), but my output settings in Filebeat.yml is output to ES, is that how it works?

Secondly, I have the visualize template and the dashboards in the kibana, I am guessing that these has caused some issues when I re-run the ./filebeat setup command and also the load template command. May I know is there a method which I can clean this situation and do a clean install ?

image

Do you have this field / data in your index? When I open the dashboards, there are many info cannot be found.

Hi, for your reference these are the fields:

  "default_field" : [
    "message",
    "tags",
    "agent.ephemeral_id",
    "agent.id",
    "agent.name",
    "agent.type",
    "agent.version",
    "client.address",
    "client.domain",
    "client.geo.city_name",
    "client.geo.continent_name",
    "client.geo.country_iso_code",
    "client.geo.country_name",
    "client.geo.name",
    "client.geo.region_iso_code",
    "client.geo.region_name",
    "client.mac",
    "client.user.email",
    "client.user.full_name",
    "client.user.group.id",
    "client.user.group.name",
    "client.user.hash",
    "client.user.id",
    "client.user.name",
    "cloud.account.id",
    "cloud.availability_zone",
    "cloud.instance.id",
    "cloud.instance.name",
    "cloud.machine.type",
    "cloud.provider",
    "cloud.region",
    "container.id",
    "container.image.name",
    "container.image.tag",
    "container.name",
    "container.runtime",
    "destination.address",
    "destination.domain",
    "destination.geo.city_name",
    "destination.geo.continent_name",
    "destination.geo.country_iso_code",
    "destination.geo.country_name",
    "destination.geo.name",
    "destination.geo.region_iso_code",
    "destination.geo.region_name",
    "destination.mac",
    "destination.user.email",
    "destination.user.full_name",
    "destination.user.group.id",
    "destination.user.group.name",
    "destination.user.hash",
    "destination.user.id",
    "destination.user.name",
    "ecs.version",
    "error.code",
    "error.id",
    "error.message",
    "event.action",
    "event.category",
    "event.dataset",
    "event.hash",
    "event.id",
    "event.kind",
    "event.module",
    "event.original",
    "event.outcome",
    "event.timezone",
    "event.type",
    "file.device",
    "file.extension",
    "file.gid",
    "file.group",
    "file.inode",
    "file.mode",
    "file.owner",
    "file.path",
    "file.target_path",
    "file.type",
    "file.uid",
    "geo.city_name",
    "geo.continent_name",
    "geo.country_iso_code",
    "geo.country_name",
    "geo.name",
    "geo.region_iso_code",
    "geo.region_name",
    "group.id",
    "group.name",
    "host.architecture",
    "host.geo.city_name",
    "host.geo.continent_name",
    "host.geo.country_iso_code",
    "host.geo.country_name",
    "host.geo.name",
    "host.geo.region_iso_code",
    "host.geo.region_name",
    "host.hostname",
    "host.id",
    "host.mac",
    "host.name",
    "host.os.family",
    "host.os.full",
    "host.os.kernel",
    "host.os.name",
    "host.os.platform",
    "host.os.version",
    "host.type",
    "host.user.email",
    "host.user.full_name",

Part 2 :

    "host.user.group.id",
    "host.user.group.name",
    "host.user.hash",
    "host.user.id",
    "host.user.name",
    "http.request.body.content",
    "http.request.method",
    "http.request.referrer",
    "http.response.body.content",
    "http.version",
    "log.level",
    "log.original",
    "network.application",
    "network.community_id",
    "network.direction",
    "network.iana_number",
    "network.name",
    "network.protocol",
    "network.transport",
    "network.type",
    "observer.geo.city_name",
    "observer.geo.continent_name",
    "observer.geo.country_iso_code",
    "observer.geo.country_name",
    "observer.geo.name",
    "observer.geo.region_iso_code",
    "observer.geo.region_name",
    "observer.hostname",
    "observer.mac",
    "observer.os.family",
    "observer.os.full",
    "observer.os.kernel",
    "observer.os.name",
    "observer.os.platform",
    "observer.os.version",
    "observer.serial_number",
    "observer.type",
    "observer.vendor",
    "observer.version",
    "organization.id",
    "organization.name",
    "os.family",
    "os.full",
    "os.kernel",
    "os.name",
    "os.platform",
    "os.version",
    "process.args",
    "process.executable",
    "process.name",
    "process.title",
    "process.working_directory",
    "server.address",
    "server.domain",
    "server.geo.city_name",
    "server.geo.continent_name",
    "server.geo.country_iso_code",
    "server.geo.country_name",
    "server.geo.name",
    "server.geo.region_iso_code",
    "server.geo.region_name",
    "server.mac",
    "server.user.email",
    "server.user.full_name",
    "server.user.group.id",
    "server.user.group.name",
    "server.user.hash",
    "server.user.id",
    "server.user.name",
    "service.ephemeral_id",
    "service.id",
    "service.name",
    "service.state",
    "service.type",
    "service.version",
    "source.address",
    "source.domain",
    "source.geo.city_name",
    "source.geo.continent_name",
    "source.geo.country_iso_code",
    "source.geo.country_name",
    "source.geo.name",
    "source.geo.region_iso_code",
    "source.geo.region_name",
    "source.mac",
    "source.user.email",
    "source.user.full_name",
    "source.user.group.id",
    "source.user.group.name",
    "source.user.hash",
    "source.user.id",
    "source.user.name",
    "url.domain",
    "url.fragment",
    "url.full",
    "url.original",
    "url.password",
    "url.path",
    "url.query",
    "url.scheme",
    "url.username",
    "user.email",
    "user.full_name",
    "user.group.id",
    "user.group.name",
    "user.hash",
    "user.id",
    "user.name",
    "user_agent.device.name",
    "user_agent.name",
    "user_agent.original",
    "user_agent.os.family",
    "user_agent.os.full",
    "user_agent.os.kernel",
    "user_agent.os.name",
    "user_agent.os.platform",
    "user_agent.os.version",
    "user_agent.version",
    "agent.hostname",
    "error.type",
    "timeseries.instance",
    "cloud.project.id",
    "cloud.image.id",
    "host.os.build",
    "host.os.codename",
    "kubernetes.pod.name",
    "kubernetes.pod.uid",
    "kubernetes.namespace",
    "kubernetes.node.name",
    "kubernetes.replicaset.name",
    "kubernetes.deployment.name",
    "kubernetes.statefulset.name",
    "kubernetes.container.name",
    "kubernetes.container.image",
    "jolokia.agent.version",
    "jolokia.agent.id",
    "jolokia.server.product",
    "jolokia.server.version",
    "jolokia.server.vendor",
    "jolokia.url",
    "log.file.path",
    "log.source.address",
    "stream",
    "input.type",
    "syslog.severity_label",
    "syslog.facility_label",
    "process.program",
    "log.flags",
    "user_agent.os.full_name",
    "fileset.name",
    "event.code",
    "icmp.code",
    "icmp.type",
    "igmp.type",
    "source.as.organization.name",
    "destination.as.organization.name",
    "apache.access.ssl.protocol",
    "apache.access.ssl.cipher",
    "apache.error.module",
    "user.terminal",
    "user.audit.id",
    "user.audit.name",
    "user.audit.group.id",
    "user.audit.group.name",

Part 3:

    "user.effective.id",
    "user.effective.name",
    "user.effective.group.id",
    "user.effective.group.name",
    "user.filesystem.id",
    "user.filesystem.name",
    "user.filesystem.group.id",
    "user.filesystem.group.name",
    "user.owner.id",
    "user.owner.name",
    "user.owner.group.id",
    "user.owner.group.name",
    "user.saved.id",
    "user.saved.name",
    "user.saved.group.id",
    "user.saved.group.name",
    "auditd.log.old_auid",
    "auditd.log.new_auid",
    "auditd.log.old_ses",
    "auditd.log.new_ses",
    "auditd.log.items",
    "auditd.log.item",
    "auditd.log.tty",
    "auditd.log.a0",
    "cisco.asa.message_id",
    "cisco.asa.suffix",
    "cisco.asa.source_interface",
    "cisco.asa.destination_interface",
    "cisco.asa.list_id",
    "cisco.asa.source_username",
    "cisco.asa.destination_username",
    "cisco.asa.threat_level",
    "cisco.asa.threat_category",
    "cisco.asa.connection_id",
    "cisco.ios.access_list",
    "cisco.ios.facility",
    "coredns.id",
    "coredns.query.class",
    "coredns.query.name",
    "coredns.query.type",
    "coredns.response.code",
    "coredns.response.flags",
    "elasticsearch.component",
    "elasticsearch.cluster.uuid",
    "elasticsearch.cluster.name",
    "elasticsearch.node.id",
    "elasticsearch.node.name",
    "elasticsearch.index.name",
    "elasticsearch.index.id",
    "elasticsearch.shard.id",
    "elasticsearch.audit.layer",
    "elasticsearch.audit.event_type",
    "elasticsearch.audit.origin.type",
    "elasticsearch.audit.realm",
    "elasticsearch.audit.user.realm",
    "elasticsearch.audit.user.roles",
    "elasticsearch.audit.action",
    "elasticsearch.audit.url.params",
    "elasticsearch.audit.indices",
    "elasticsearch.audit.request.id",
    "elasticsearch.audit.request.name",
    "elasticsearch.audit.message",
    "elasticsearch.gc.phase.name",
    "elasticsearch.gc.tags",
    "elasticsearch.slowlog.logger",
    "elasticsearch.slowlog.took",
    "elasticsearch.slowlog.types",
    "elasticsearch.slowlog.stats",
    "elasticsearch.slowlog.search_type",
    "elasticsearch.slowlog.source_query",
    "elasticsearch.slowlog.extra_source",
    "elasticsearch.slowlog.total_hits",
    "elasticsearch.slowlog.total_shards",
    "elasticsearch.slowlog.routing",
    "elasticsearch.slowlog.id",
    "elasticsearch.slowlog.type",
    "envoyproxy.log_type",
    "envoyproxy.response_flags",
    "envoyproxy.request_id",
    "envoyproxy.authority",
    "envoyproxy.proxy_type",
    "googlecloud.vpcflow.reporter",
    "googlecloud.vpcflow.destination.instance.project_id",
    "googlecloud.vpcflow.destination.instance.region",
    "googlecloud.vpcflow.destination.instance.zone",
    "googlecloud.vpcflow.destination.vpc.project_id",
    "googlecloud.vpcflow.destination.vpc.vpc_name",
    "googlecloud.vpcflow.destination.vpc.subnetwork_name",
    "googlecloud.vpcflow.source.instance.project_id",
    "googlecloud.vpcflow.source.instance.region",
    "googlecloud.vpcflow.source.instance.zone",
    "googlecloud.vpcflow.source.vpc.project_id",
    "googlecloud.vpcflow.source.vpc.vpc_name",
    "googlecloud.vpcflow.source.vpc.subnetwork_name",
    "haproxy.frontend_name",
    "haproxy.backend_name",
    "haproxy.server_name",
    "haproxy.bind_name",
    "haproxy.error_message",
    "haproxy.source",
    "haproxy.termination_state",
    "haproxy.mode",
    "haproxy.http.response.captured_cookie",
    "haproxy.http.response.captured_headers",
    "haproxy.http.request.captured_cookie",
    "haproxy.http.request.captured_headers",
    "haproxy.http.request.raw_request_line",
    "icinga.debug.facility",
    "icinga.main.facility",
    "icinga.startup.facility",
    "iis.access.site_name",
    "iis.access.server_name",
    "iis.access.cookie",
    "iis.error.reason_phrase",
    "iis.error.queue_name",
    "iptables.fragment_flags",
    "iptables.input_device",
    "iptables.output_device",
    "iptables.tcp.flags",
    "iptables.ubiquiti.input_zone",
    "iptables.ubiquiti.output_zone",
    "iptables.ubiquiti.rule_number",
    "iptables.ubiquiti.rule_set",
    "kafka.log.component",
    "kafka.log.class",
    "kafka.log.trace.class",
    "kafka.log.trace.message",
    "kibana.log.tags",
    "kibana.log.state",
    "logstash.log.module",
    "text",
    "logstash.log.thread",
    "logstash.slowlog.module",
    "text",
    "logstash.slowlog.thread",
    "text",
    "logstash.slowlog.event",
    "logstash.slowlog.plugin_name",
    "logstash.slowlog.plugin_type",
    "text",
    "logstash.slowlog.plugin_params",
    "mongodb.log.component",
    "mongodb.log.context",
    "mssql.log.origin",
    "mysql.slowlog.query",
    "mysql.slowlog.schema",
    "mysql.slowlog.current_user",
    "mysql.slowlog.last_errno",
    "mysql.slowlog.killed",
    "mysql.slowlog.log_slow_rate_type",
    "mysql.slowlog.log_slow_rate_limit",
    "mysql.slowlog.innodb.trx_id",
    "nats.log.msg.type",
    "nats.log.msg.subject",
    "nats.log.msg.reply_to",
    "nats.log.msg.error.message",
    "nats.log.msg.queue_group",
    "osquery.result.name",
    "osquery.result.action",
    "osquery.result.host_identifier",
    "osquery.result.calendar_time",
    "panw.panos.ruleset",
    "panw.panos.source.zone",
    "panw.panos.source.interface",

Part 4: Final

    "panw.panos.destination.zone",
    "panw.panos.destination.interface",
    "panw.panos.network.pcap_id",
    "panw.panos.network.nat.community_id",
    "panw.panos.file.hash",
    "panw.panos.url.category",
    "panw.panos.flow_id",
    "panw.panos.threat.resource",
    "panw.panos.threat.id",
    "panw.panos.threat.name",
    "postgresql.log.timestamp",
    "postgresql.log.database",
    "postgresql.log.query",
    "rabbitmq.log.pid",
    "redis.log.role",
    "redis.slowlog.cmd",
    "redis.slowlog.key",
    "redis.slowlog.args",
    "santa.action",
    "santa.decision",
    "santa.reason",
    "santa.mode",
    "santa.disk.volume",
    "santa.disk.bus",
    "santa.disk.serial",
    "santa.disk.bsdname",
    "santa.disk.model",
    "santa.disk.fs",
    "santa.disk.mount",
    "certificate.common_name",
    "certificate.sha256",
    "hash.sha256",
    "suricata.eve.event_type",
    "suricata.eve.app_proto_orig",
    "suricata.eve.tcp.tcp_flags",
    "suricata.eve.tcp.tcp_flags_tc",
    "suricata.eve.tcp.state",
    "suricata.eve.tcp.tcp_flags_ts",
    "suricata.eve.fileinfo.sha1",
    "suricata.eve.fileinfo.state",
    "suricata.eve.fileinfo.sha256",
    "suricata.eve.fileinfo.md5",
    "suricata.eve.dns.type",
    "suricata.eve.dns.rrtype",
    "suricata.eve.dns.rrname",
    "suricata.eve.dns.rdata",
    "suricata.eve.dns.rcode",
    "suricata.eve.flow_id",
    "suricata.eve.email.status",
    "suricata.eve.http.redirect",
    "suricata.eve.http.protocol",
    "suricata.eve.http.http_content_type",
    "suricata.eve.in_iface",
    "suricata.eve.alert.category",
    "suricata.eve.alert.signature",
    "suricata.eve.ssh.client.proto_version",
    "suricata.eve.ssh.client.software_version",
    "suricata.eve.ssh.server.proto_version",
    "suricata.eve.ssh.server.software_version",
    "suricata.eve.tls.issuerdn",
    "suricata.eve.tls.sni",
    "suricata.eve.tls.version",
    "suricata.eve.tls.fingerprint",
    "suricata.eve.tls.serial",
    "suricata.eve.tls.subject",
    "suricata.eve.app_proto_ts",
    "suricata.eve.flow.state",
    "suricata.eve.flow.reason",
    "suricata.eve.app_proto_tc",
    "suricata.eve.smtp.rcpt_to",
    "suricata.eve.smtp.mail_from",
    "suricata.eve.smtp.helo",
    "suricata.eve.app_proto_expected",
    "system.auth.ssh.method",
    "system.auth.ssh.signature",
    "system.auth.ssh.event",
    "system.auth.sudo.error",
    "system.auth.sudo.tty",
    "system.auth.sudo.pwd",
    "system.auth.sudo.user",
    "system.auth.sudo.command",
    "system.auth.useradd.home",
    "system.auth.useradd.shell",
    "traefik.access.user_identifier",
    "traefik.access.frontend_name",
    "traefik.access.backend_url",
    "zeek.session_id",
    "zeek.connection.state",
    "zeek.connection.history",
    "zeek.connection.orig_l2_addr",
    "zeek.connection.resp_l2_addr",
    "zeek.dns.trans_id",
    "zeek.dns.query",
    "zeek.dns.qclass_name",
    "zeek.dns.qtype_name",
    "zeek.dns.rcode_name",
    "zeek.dns.answers",
    "zeek.http.status_msg",
    "zeek.http.info_msg",
    "zeek.http.tags",
    "zeek.http.password",
    "zeek.http.proxied",
    "zeek.http.client_header_names",
    "zeek.http.server_header_names",
    "zeek.http.orig_fuids",
    "zeek.http.orig_mime_types",
    "zeek.http.orig_filenames",
    "zeek.http.resp_fuids",
    "zeek.http.resp_mime_types",
    "zeek.http.resp_filenames",
    "zeek.files.fuid",
    "zeek.files.session_ids",
    "zeek.files.source",
    "zeek.files.analyzers",
    "zeek.files.mime_type",
    "zeek.files.filename",
    "zeek.files.parent_fuid",
    "zeek.files.md5",
    "zeek.files.sha1",
    "zeek.files.sha256",
    "zeek.files.extracted",
    "zeek.ssl.version",
    "zeek.ssl.cipher",
    "zeek.ssl.curve",
    "zeek.ssl.server_name",
    "zeek.ssl.next_protocol",
    "zeek.ssl.cert_chain",
    "zeek.ssl.cert_chain_fuids",
    "zeek.ssl.client_cert_chain",
    "zeek.ssl.client_cert_chain_fuids",
    "zeek.ssl.issuer",
    "zeek.ssl.client_issuer",
    "zeek.ssl.validation_status",
    "zeek.ssl.validation_code",
    "zeek.ssl.subject",
    "zeek.ssl.client_subject",
    "zeek.ssl.last_alert",
    "zeek.notice.connection_id",
    "zeek.notice.icmp_id",
    "zeek.notice.file.id",
    "zeek.notice.file.parent_id",
    "zeek.notice.file.source",
    "zeek.notice.file.mime_type",
    "zeek.notice.fuid",
    "zeek.notice.note",
    "zeek.notice.msg",
    "zeek.notice.sub",
    "zeek.notice.peer_name",
    "zeek.notice.peer_descr",
    "zeek.notice.actions",
    "zeek.notice.email_body_sections",
    "zeek.notice.email_delay_tokens",
    "zeek.notice.identifier",
    "fields.*"

Did that come from?

GET /_template/filebeat-7.1.1

If so further down you will see this.......which is correct. The out put is very long.

   "source" : {
      "properties" : {
        "geo" : {
          "properties" : {
            "region_iso_code" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "continent_name" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "city_name" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "country_iso_code" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "country_name" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "name" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "region_name" : {
              "ignore_above" : 1024,
              "type" : "keyword"
            },
            "location" : {
              "type" : "geo_point"
            }
          }
        },

In the index pattern you should see these fields like this shown below...

if not delete that index pattern from the Kibana GUI
Mangement -> Index Patterns
and run ./filebeat setup again.

Apologies but I don't think I am able to help much more ... to do a clean install ... uninstall and re-install elasticsearch and make sure the the data directory under the elasticsearch install is removed before you reinstall.

Hi may I check with you. My client side is installing filebeat 7.3.0 while the ES server's Filebeat is on 7.1.1. Will there be an issue in terms of compatibility and the data format?

Hi @stephenb, I manage to get this fields, but the dashboard still cannot capture.

What I did is I removed the client Filebeat and installed the 7.1.1 version.

image

After that I notice I am facing some difficulties installing the logstash on the client server, its Centos 6.10 and with Java 1.8.0. Will there be any problem connecting these 2 servers ?

Hi is there anyone can help to troubleshoot this issue? thx

@stephenb , may I know what is your index name for this Nginx elastic ?

I came across some forum suggest that the template only recognize nginx-* indexes in order to load those templates.