Configuring Filebeat to pack openresty/nginx logs and visualize in Kibana dashboards

[kpagcha]

I am totally new with the ELK stack and not really a sysadmin either, just a web developer trying to figure this out.

I have two droplets:

• one where I installed the ELK stack successfully to some extent (managed to visualize syslogs with the filebeat system module)
• another where my openresty web server works. openresty is an nginx distribution.

I managed to configure filebeat in the openresty droplet to send the logs:

...
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /home/scheduler/logs/openresty/access.log
    - /home/scheduler/logs/openresty/error.log
  fields:
    type: nginx

...
# commented out Elasticsearch output
#output.elasticsearch:
...
output.logstash:
  hosts: ["<ELK_DROPLET_PUBLIC_IP>:5044"]
...

And in the ELK droplet I have this logstash configuration file:

input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "openresty-logs-%{+YYYY.MM.dd}"
    pipeline => "openresty_log_type_pipeline"
  }
}

This works and I have even set up a pipeline through Kibana to classify access/error entries.

Now, I am trying to get the Kibana dashboards for nginx to work with my openresty logs but after much research and effort I have hit a wall.

I have enabled the nginx module in the openresty droplet:

sudo filebeat modules enable nginx

And configured the filebeat module to use the openresty logs:

# /etc/filebeat/modules.d/nginx.yml
- module: nginx
  access:
    enabled: true
    var.paths: ["/home/scheduler/logs/openresty/access.log"]

  error:
    enabled: true
    var.paths: ["/home/scheduler/logs/openresty/error.log"]

  ingress_controller:
    enabled: false

But I am not even sure if this is working. Is there a way to tell whether the nginx data is being shipped with filebeat to logstash in the other droplet? Modules are imported by default in the filebeat config.

I am aware that I will probably need to take care of proper log formatting. But the first worry is not being able to get the openresty/nginx logs to Logstash and indexed properly for the dashboard to be able to take the data.

What do I need to do to have the Kibana nginx dashboards work with my openresty logs?

[jsoriano (Jaime Soriano)]

Hey @kpagcha, welcome to discuss

If you are starting with the Elastic Stack now, let me give you a couple of tips

First, you probably don't need Logstash, Beats can push data directly to Elasticsearch. Logstash is great when you need advanced processing or you have complex architectures, but at this point you probably don't have any of them.

Second, take a look to Fleet. With it you only need to install a single agent on any of your droplets, and then use Kibana to manage the specific configurations you need on each one of them. Under the hood, Fleet and Agent also use Beats. And you can find the system and nginx integrations also in Fleet.
Take a look here: Fleet and Elastic Agent overview | Fleet and Elastic Agent Guide [8.9] | Elastic

If you want to continue using Filebeat directly without Fleet, I would still recommend you to use the elasticsearch output directly, without Logstash. It will be easier to start with.

[kpagcha]

Thank you, I'll have a look.

So you're suggesting I skip logstash and send directly to elasticsearch? For that I'd comment out the logstash block in the filebeat.yml file right? Would the index be created automatically? And most importantly how do I plugin in the nginx Kibana dashboard? This is the same issue I had before.

[jsoriano (Jaime Soriano)]

If connected directly to Elasticearch, Filebeat will take care of creating the index, yes. And also the ingest pipelines.

For the dashboard, if it is installed and you are ingesting data directly with Filebeat, it should work. If not, what is the problem you are having? Is it not showing any data?

If you haven't installed it, remember to run filebeat setup. Find docs here.

[kpagcha]

I have configured filebeat to point to Elasticsearch and Kibana on the corresponding server, have ports 9200 and 5601 listening and changed networks to 0.0.0.0. Filebeat seems to be properly connected.

It seems Filebeat is finally working, I can see updates in the index:

yellow open   filebeat-7.17.11-2023.07.26-000001 AXJ_ISJyQTyspVe5w6JxfA   1   1     194682            0     45.8mb         45.8mb

However, the dashboard shows empty data: "no results found".