Hi all, i found that this Nginx Module's visualization does not provides the full template. How can I get it and steps to install it?
As when I view the dashboard, some are appeared "Could not locate that index-pattern (id: filebeat-*), click here to re-create it"
hi @skyluke.1987, it looks like the dashboards have been loaded but the index pattern is missing.
Can you check if the filebeat-* index pattern exists or have you configured a custom name for the filebeat index?
These are the usual steps on loading the dashboards https://www.elastic.co/guide/en/beats/filebeat/current/load-kibana-dashboards.html
Hi @MarianaD, the index filebeat-* has been loaded, and the name was "filebeat-7.3.0" does this index naming will impact the creation on the dashboard?
Hi I found this suggestion saying we should removed all the "saved objects" and reimport the items again. But my concern in there are other saved objects which are not owned by this filebeat nginx module. How should I do?
Hi can anyone help me on this ?
Firstly, I have difficulties create multiple filebeat files by date. Even I have specify in the Filebeat>modules.d>nginx.yml file with the export filename.
Next, regarding the Nginx module, I am still not able to get the dashboard working.
Did you run filebeat setup after you enabled the nginx module? setup loads the index template, index pattern and all the visualizations etc
Hi yes, I did. Below is a screenshot of it.
Hi MarianaD, what if I created the index pattern name this way filebeat-7.3.0* ? Will this affects the result?
I notice I was running Filebeat 7.1 instead of the new Filebeat7.3.. How can I choose to run the new one ? I remembered i have installed the new one before.
Hi there, is there anyone encounter this problem and have solution to it?
Can I know how can I print out the data / fields received, does it contain the required fields, e.g. geoip.
How can I control the data fields, and what if there is no field captured, what should I do? Currently the visualization showing empty. Error: "Could not locate that index-pattern-field (id: source.address)" and more.
Apologies but It's a bit difficult to know what state your whole system is in but if you're all on 7.3 then you could try to delete the index pattern in kibana and then run filebeat setup again with the nginx module enabled and it should create the correct index pattern.
However if you have data from old filebeats and new filebeats they may not all work with the new index pattern.
I started from scratch installed filebeat and enabled nginx module ran setup and then send directly to elasticsearch all the visualizations and dashboards load. That's not to say that you're not running into issues but from a clean configuration it should work
Hi @stephenb, thanks for your reply and analysis. Currently I am using version 7.1 with Kibana and Elasticsearch. I notice that the visualizations cannot find the correct indices and this has caused the template cannot be loaded. May I know is there a way where we can list out all the fields and from there re-link all the required fields.
Secondly, I notice that on the other end (The Nginx server) the log files (.access and .error) log files contain very little information. Things like locations all these are not available. This could be one of the reason why my Map's visualization cannot be loaded with "No Data Found".
Kindly advise and share your thoughts.
Hi all, if I perform a "filebeat export template > test.json" and the file showing there are fields that are required, e.g geo.location etc. But when I open the visualization, it prompted me "No Data". Why is it so ?
filebeat export template > test.json
This command shows what template will be loaded when it is run not what IS currently loaded in the Elasticsearch cluster.
If you want to see what is currently loaded use this
curl http://localhost:9200/_template/filebeat-7.1.1
Or go to the Dev Tools and Run.
GET /_template/filebeat-7.1.1
Also confusing to me is that some of your screen shots show logstash are you using logstash as well? I would first get the simple Filebeat -> Elasticsearch directly.
The screen shot above shows the logstash output not filebeat setup that is a little more complex to setup.
Me if I were you I would start with a clean setup or you will need to remove the template, index patterns and existing filebeat indexes
Just to test it out...
I just built a brand new 7.1.1 single node Elasticsearch and Kibana on localhost and everything works fine the first time. I did not use Logstash.
I simply started Elasticsearch and Kibana without editing any settings.
Enabled ngnix module
./filebeat modules enable nginx
ran setup
./filebeat setup
Downloaded the nginx example logs file (see below for link).
edit the modules/nginx.yml and set the path to the nginx log file I just downloaded.
# Module: nginx
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.1/filebeat-module-nginx.html
- module: nginx
# Access logs
access:
enabled: true
# Set custom paths for the log files. If left empty,
# Filebeat will choose the paths depending on your OS.
var.paths: [ "/Users/sbrown/Downloads/nginx_logs.log" ]
Then started filebeat
./filebeat -e
The data loaded and the visualizations work fine with 1 exception there are no error logs so those are blank, this data set only contains access logs. The data is from May 2015.
Here is the data I loaded....
Hi @stephenb, thanks for your detailed reply. Because of my setup involved these 2 different servers (A & B), but my output settings in Filebeat.yml is output to ES, is that how it works?
Secondly, I have the visualize template and the dashboards in the kibana, I am guessing that these has caused some issues when I re-run the ./filebeat setup command and also the load template command. May I know is there a method which I can clean this situation and do a clean install ?
Do you have this field / data in your index? When I open the dashboards, there are many info cannot be found.
Hi, for your reference these are the fields:
"default_field" : [
"message",
"tags",
"agent.ephemeral_id",
"agent.id",
"agent.name",
"agent.type",
"agent.version",
"client.address",
"client.domain",
"client.geo.city_name",
"client.geo.continent_name",
"client.geo.country_iso_code",
"client.geo.country_name",
"client.geo.name",
"client.geo.region_iso_code",
"client.geo.region_name",
"client.mac",
"client.user.email",
"client.user.full_name",
"client.user.group.id",
"client.user.group.name",
"client.user.hash",
"client.user.id",
"client.user.name",
"cloud.account.id",
"cloud.availability_zone",
"cloud.instance.id",
"cloud.instance.name",
"cloud.machine.type",
"cloud.provider",
"cloud.region",
"container.id",
"container.image.name",
"container.image.tag",
"container.name",
"container.runtime",
"destination.address",
"destination.domain",
"destination.geo.city_name",
"destination.geo.continent_name",
"destination.geo.country_iso_code",
"destination.geo.country_name",
"destination.geo.name",
"destination.geo.region_iso_code",
"destination.geo.region_name",
"destination.mac",
"destination.user.email",
"destination.user.full_name",
"destination.user.group.id",
"destination.user.group.name",
"destination.user.hash",
"destination.user.id",
"destination.user.name",
"ecs.version",
"error.code",
"error.id",
"error.message",
"event.action",
"event.category",
"event.dataset",
"event.hash",
"event.id",
"event.kind",
"event.module",
"event.original",
"event.outcome",
"event.timezone",
"event.type",
"file.device",
"file.extension",
"file.gid",
"file.group",
"file.inode",
"file.mode",
"file.owner",
"file.path",
"file.target_path",
"file.type",
"file.uid",
"geo.city_name",
"geo.continent_name",
"geo.country_iso_code",
"geo.country_name",
"geo.name",
"geo.region_iso_code",
"geo.region_name",
"group.id",
"group.name",
"host.architecture",
"host.geo.city_name",
"host.geo.continent_name",
"host.geo.country_iso_code",
"host.geo.country_name",
"host.geo.name",
"host.geo.region_iso_code",
"host.geo.region_name",
"host.hostname",
"host.id",
"host.mac",
"host.name",
"host.os.family",
"host.os.full",
"host.os.kernel",
"host.os.name",
"host.os.platform",
"host.os.version",
"host.type",
"host.user.email",
"host.user.full_name",Part 2 :
"host.user.group.id",
"host.user.group.name",
"host.user.hash",
"host.user.id",
"host.user.name",
"http.request.body.content",
"http.request.method",
"http.request.referrer",
"http.response.body.content",
"http.version",
"log.level",
"log.original",
"network.application",
"network.community_id",
"network.direction",
"network.iana_number",
"network.name",
"network.protocol",
"network.transport",
"network.type",
"observer.geo.city_name",
"observer.geo.continent_name",
"observer.geo.country_iso_code",
"observer.geo.country_name",
"observer.geo.name",
"observer.geo.region_iso_code",
"observer.geo.region_name",
"observer.hostname",
"observer.mac",
"observer.os.family",
"observer.os.full",
"observer.os.kernel",
"observer.os.name",
"observer.os.platform",
"observer.os.version",
"observer.serial_number",
"observer.type",
"observer.vendor",
"observer.version",
"organization.id",
"organization.name",
"os.family",
"os.full",
"os.kernel",
"os.name",
"os.platform",
"os.version",
"process.args",
"process.executable",
"process.name",
"process.title",
"process.working_directory",
"server.address",
"server.domain",
"server.geo.city_name",
"server.geo.continent_name",
"server.geo.country_iso_code",
"server.geo.country_name",
"server.geo.name",
"server.geo.region_iso_code",
"server.geo.region_name",
"server.mac",
"server.user.email",
"server.user.full_name",
"server.user.group.id",
"server.user.group.name",
"server.user.hash",
"server.user.id",
"server.user.name",
"service.ephemeral_id",
"service.id",
"service.name",
"service.state",
"service.type",
"service.version",
"source.address",
"source.domain",
"source.geo.city_name",
"source.geo.continent_name",
"source.geo.country_iso_code",
"source.geo.country_name",
"source.geo.name",
"source.geo.region_iso_code",
"source.geo.region_name",
"source.mac",
"source.user.email",
"source.user.full_name",
"source.user.group.id",
"source.user.group.name",
"source.user.hash",
"source.user.id",
"source.user.name",
"url.domain",
"url.fragment",
"url.full",
"url.original",
"url.password",
"url.path",
"url.query",
"url.scheme",
"url.username",
"user.email",
"user.full_name",
"user.group.id",
"user.group.name",
"user.hash",
"user.id",
"user.name",
"user_agent.device.name",
"user_agent.name",
"user_agent.original",
"user_agent.os.family",
"user_agent.os.full",
"user_agent.os.kernel",
"user_agent.os.name",
"user_agent.os.platform",
"user_agent.os.version",
"user_agent.version",
"agent.hostname",
"error.type",
"timeseries.instance",
"cloud.project.id",
"cloud.image.id",
"host.os.build",
"host.os.codename",
"kubernetes.pod.name",
"kubernetes.pod.uid",
"kubernetes.namespace",
"kubernetes.node.name",
"kubernetes.replicaset.name",
"kubernetes.deployment.name",
"kubernetes.statefulset.name",
"kubernetes.container.name",
"kubernetes.container.image",
"jolokia.agent.version",
"jolokia.agent.id",
"jolokia.server.product",
"jolokia.server.version",
"jolokia.server.vendor",
"jolokia.url",
"log.file.path",
"log.source.address",
"stream",
"input.type",
"syslog.severity_label",
"syslog.facility_label",
"process.program",
"log.flags",
"user_agent.os.full_name",
"fileset.name",
"event.code",
"icmp.code",
"icmp.type",
"igmp.type",
"source.as.organization.name",
"destination.as.organization.name",
"apache.access.ssl.protocol",
"apache.access.ssl.cipher",
"apache.error.module",
"user.terminal",
"user.audit.id",
"user.audit.name",
"user.audit.group.id",
"user.audit.group.name",Part 3:
"user.effective.id",
"user.effective.name",
"user.effective.group.id",
"user.effective.group.name",
"user.filesystem.id",
"user.filesystem.name",
"user.filesystem.group.id",
"user.filesystem.group.name",
"user.owner.id",
"user.owner.name",
"user.owner.group.id",
"user.owner.group.name",
"user.saved.id",
"user.saved.name",
"user.saved.group.id",
"user.saved.group.name",
"auditd.log.old_auid",
"auditd.log.new_auid",
"auditd.log.old_ses",
"auditd.log.new_ses",
"auditd.log.items",
"auditd.log.item",
"auditd.log.tty",
"auditd.log.a0",
"cisco.asa.message_id",
"cisco.asa.suffix",
"cisco.asa.source_interface",
"cisco.asa.destination_interface",
"cisco.asa.list_id",
"cisco.asa.source_username",
"cisco.asa.destination_username",
"cisco.asa.threat_level",
"cisco.asa.threat_category",
"cisco.asa.connection_id",
"cisco.ios.access_list",
"cisco.ios.facility",
"coredns.id",
"coredns.query.class",
"coredns.query.name",
"coredns.query.type",
"coredns.response.code",
"coredns.response.flags",
"elasticsearch.component",
"elasticsearch.cluster.uuid",
"elasticsearch.cluster.name",
"elasticsearch.node.id",
"elasticsearch.node.name",
"elasticsearch.index.name",
"elasticsearch.index.id",
"elasticsearch.shard.id",
"elasticsearch.audit.layer",
"elasticsearch.audit.event_type",
"elasticsearch.audit.origin.type",
"elasticsearch.audit.realm",
"elasticsearch.audit.user.realm",
"elasticsearch.audit.user.roles",
"elasticsearch.audit.action",
"elasticsearch.audit.url.params",
"elasticsearch.audit.indices",
"elasticsearch.audit.request.id",
"elasticsearch.audit.request.name",
"elasticsearch.audit.message",
"elasticsearch.gc.phase.name",
"elasticsearch.gc.tags",
"elasticsearch.slowlog.logger",
"elasticsearch.slowlog.took",
"elasticsearch.slowlog.types",
"elasticsearch.slowlog.stats",
"elasticsearch.slowlog.search_type",
"elasticsearch.slowlog.source_query",
"elasticsearch.slowlog.extra_source",
"elasticsearch.slowlog.total_hits",
"elasticsearch.slowlog.total_shards",
"elasticsearch.slowlog.routing",
"elasticsearch.slowlog.id",
"elasticsearch.slowlog.type",
"envoyproxy.log_type",
"envoyproxy.response_flags",
"envoyproxy.request_id",
"envoyproxy.authority",
"envoyproxy.proxy_type",
"googlecloud.vpcflow.reporter",
"googlecloud.vpcflow.destination.instance.project_id",
"googlecloud.vpcflow.destination.instance.region",
"googlecloud.vpcflow.destination.instance.zone",
"googlecloud.vpcflow.destination.vpc.project_id",
"googlecloud.vpcflow.destination.vpc.vpc_name",
"googlecloud.vpcflow.destination.vpc.subnetwork_name",
"googlecloud.vpcflow.source.instance.project_id",
"googlecloud.vpcflow.source.instance.region",
"googlecloud.vpcflow.source.instance.zone",
"googlecloud.vpcflow.source.vpc.project_id",
"googlecloud.vpcflow.source.vpc.vpc_name",
"googlecloud.vpcflow.source.vpc.subnetwork_name",
"haproxy.frontend_name",
"haproxy.backend_name",
"haproxy.server_name",
"haproxy.bind_name",
"haproxy.error_message",
"haproxy.source",
"haproxy.termination_state",
"haproxy.mode",
"haproxy.http.response.captured_cookie",
"haproxy.http.response.captured_headers",
"haproxy.http.request.captured_cookie",
"haproxy.http.request.captured_headers",
"haproxy.http.request.raw_request_line",
"icinga.debug.facility",
"icinga.main.facility",
"icinga.startup.facility",
"iis.access.site_name",
"iis.access.server_name",
"iis.access.cookie",
"iis.error.reason_phrase",
"iis.error.queue_name",
"iptables.fragment_flags",
"iptables.input_device",
"iptables.output_device",
"iptables.tcp.flags",
"iptables.ubiquiti.input_zone",
"iptables.ubiquiti.output_zone",
"iptables.ubiquiti.rule_number",
"iptables.ubiquiti.rule_set",
"kafka.log.component",
"kafka.log.class",
"kafka.log.trace.class",
"kafka.log.trace.message",
"kibana.log.tags",
"kibana.log.state",
"logstash.log.module",
"text",
"logstash.log.thread",
"logstash.slowlog.module",
"text",
"logstash.slowlog.thread",
"text",
"logstash.slowlog.event",
"logstash.slowlog.plugin_name",
"logstash.slowlog.plugin_type",
"text",
"logstash.slowlog.plugin_params",
"mongodb.log.component",
"mongodb.log.context",
"mssql.log.origin",
"mysql.slowlog.query",
"mysql.slowlog.schema",
"mysql.slowlog.current_user",
"mysql.slowlog.last_errno",
"mysql.slowlog.killed",
"mysql.slowlog.log_slow_rate_type",
"mysql.slowlog.log_slow_rate_limit",
"mysql.slowlog.innodb.trx_id",
"nats.log.msg.type",
"nats.log.msg.subject",
"nats.log.msg.reply_to",
"nats.log.msg.error.message",
"nats.log.msg.queue_group",
"osquery.result.name",
"osquery.result.action",
"osquery.result.host_identifier",
"osquery.result.calendar_time",
"panw.panos.ruleset",
"panw.panos.source.zone",
"panw.panos.source.interface",© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.