Filebeat TCP input with Nginx Module

Elastic Stack Beats
1

Hi ,

My setup:
Elasticsearch & Kibana - 7.5.2
Filebeat - 7.5.x

I have a Filebeat(NGINX-1) listen on TCP input to recieve Proxy logs from a remote NGINX server(NGINX-2) sending logs through Logstash TCP output.
My question is how to use a Filebeat.input : TCP method to ingest logs from NGINX-2 (remote proxy) and parse the logs within the same filebeat index using the currently running"nginx" module.

Also NGINX2 cannot send logs directly to Elasticsearch nodes.

I know how to configure logstash and parse the logs separately, but i would like to explore using one filebeat tcp input with nginx module enabled.

2

Hi @karnamonkster!

I'm not sure if I completely understand your question of what you are trying to achieve.
However, you can always use Filebeat's nginx module to parse the logs of an nginx server (from /var/log/nginx.log for instance) and ship them directly to Elasticsearch. This requires that to install Filebeat agent on the same machine with the nginx server so as Filebeat to be able to access the logs' file.

3

@ChrsMark

Let me clarify the case:
I have 2 NGINX servers.
One of them(NGINX-1) can reach Elasticsearch nodes and is sending the Nginx data perfectly fine through Filebeat

The other one (NGINX-2) cannot reach Elasticsearch node and can only communicate to NGINX-1 to send logs on a given port (say 5044) .

So i wanted to know if NGINX-1 filebeat input as TCP can consume the logs and parse the NGINX-2 logs as well.

4

Hi @ChrsMark

I am able to send the logs through TCP input to the filebeat
But is there a way to parse the message field in ECS format similar to nginx module

5

Hi!

I don't think you redirect the tcp input to a module right now. You can always use processors to customize your fields. Script processor might be handy here.

In addition, I would try to redirect the logs I receive to a file and then have a second Filebeat collecting from this file using nginx module.

C.

6

Hi @ChrsMark ,

I have managed to push the logs from the remote Nginx to this one using rsyslog.
But all i need where i can specify a GROK pattern, I remember there used to be a GROK processor, but not sure how do i use that with my filebeat.input (TCP) block.

I will look at the script processor as suggested.

7

Hi @ChrsMark,

I finally managed to get what i wanted.
Didn't know it was that easy.

  1. Moved my remote NGINX-2logs to a folder on NGINX-1 Machine say at

/var/log/NGINX2/access.log
/var/log/NGINX2/error.log

  1. Now since the NGINX modules only checks on the default path as configured in the nginx module manifest.yml located at:

    /usr/share/filebeat/module/nginx/access
    /usr/share/filebeat/module/nginx/error

  2. So made a slight change by adding the NGINX2 log paths under default path variable.
    And that was it.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.