AWS NLB config infront of FileBeats Instance

Elastic Stack Beats
1

Hi All,

I'm trying to setup a AWS Network Load Balancer (NLB) in-front of Filebeat instance. Need help in 1. How to configure the NLB &
2. How to do health check in NLB.

Note:
This setup is to collect the logs from various devices (over UDP) into Elastic Search via Filebeat agent.
Devices --> NLB--> FileBeat --> ES

Thanks,
Ravi

2

This seems more like a question for aws not elasticsearch

3

I have a query related to the FileBeat server running on the EC2 (target).

The NLB health checks are configured as TCP. Is the filebeat is capable of handling TCP health checks originating from the NLB ?

4

As answered in the other post, I think that your problem is that the AWS NLB can't do a health check using UDP, so you will need to do the health check using TCP or HTTP/HTTPS.

You can do that adding a TCP input in filebeat that will be used only for this healthcheck, than in your target group you configure the health check to use this port.

Or you can try this experimental feature that exposes the metrics http endpoint, and use this endpoint as the target for the health check.

1 Like
5

Thank you for your response - @leandrojmp

I tried the first option - configuring additional TCP input on port 9001. But, no luck - still TG is in unhealthy state. Would you please have a look at my setup below?

Filebeat Config:
#------------------------------ Udp input --------------------------------
filebeat.inputs:

  • type: udp
    enabled: true
    host: "localhost:53"
    max_message_size: 10KiB

#------------------------------ TCP input --------------------------------

  • type: tcp
    enabled: true
    host: "localhost:9001"
    max_connections: 0
    timeout: 300s

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Filebeat Startup Log:
2021-04-07T16:53:20.053+0100 INFO [crawler] beater/crawler.go:71 Loading Inputs: 3
2021-04-07T16:53:20.053+0100 DEBUG [registrar] registrar/registrar.go:140 Starting Registrar
2021-04-07T16:53:20.054+0100 INFO [crawler] beater/crawler.go:141 Starting input (ID: 10329058212159632590)
2021-04-07T16:53:20.054+0100 INFO udp/input.go:99 Starting UDP input
2021-04-07T16:53:20.054+0100 INFO [UDP] dgram/server.go:96 Started listening for UDP connection
2021-04-07T16:53:20.054+0100 INFO [crawler] beater/crawler.go:141 Starting input (ID: 6776719849853723096)
2021-04-07T16:53:20.054+0100 DEBUG [cfgfile] cfgfile/reload.go:132 Checking module configs from: /etc/filebeat/modules.d/*.yml
2021-04-07T16:53:20.054+0100 DEBUG [cfgfile] cfgfile/reload.go:146 Number of module configs found: 0
2021-04-07T16:53:20.054+0100 INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 2
2021-04-07T16:53:20.054+0100 INFO [input.tcp] tcp/input.go:106 Starting TCP input {"address": "localhost:9001"}
2021-04-07T16:53:20.055+0100 INFO cfgfile/reload.go:164 Config reloader started
2021-04-07T16:53:20.055+0100 DEBUG [cfgfile] cfgfile/reload.go:194 Scan for new config files
2021-04-07T16:53:20.055+0100 DEBUG [cfgfile] cfgfile/reload.go:213 Number of module configs found: 0
2021-04-07T16:53:20.055+0100 DEBUG [reload] cfgfile/list.go:63 Starting reload procedure, current runners: 0
2021-04-07T16:53:20.055+0100 DEBUG [reload] cfgfile/list.go:81 Start list: 0, Stop list: 0
2021-04-07T16:53:20.055+0100 INFO cfgfile/reload.go:224 Loading of config files completed.
2021-04-07T16:53:20.055+0100 INFO [tcp] streaming/listener.go:73 Started listening for TCP connection {"address": "localhost:9001"}
2021-04-07T16:53:30.055+0100 DEBUG [input] input/input.go:139 Run input

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Checked the listen port in unix:
[root@d1entesttlsr001 ~]# netstat -tulpn | grep LISTEN
tcp 0 0 127.0.0.1:9001 0.0.0.0:* LISTEN 24620/filebeat

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

NLB Health check settings:
Protocol : TCP
Port : 9001
Status : unhealthy
Status details : Health checks failed

Much appreciated your help.

Thanks,
Ravi

6

You can't listen on only localhost, you need to change the host to 0.0.0.0:9001, the same thing applies to your UDP input.

If you want to listen for connections from outside your machine, even from the same network, you can't use localhost as it will only listen to local requests. The NLB won't be able to talk to your filebeat if you are listening only on localhost.

This is not an Filebeat issue, it is an AWS issue, there is not much to do on the filebeat side.

1 Like
7

Thanks a lot @leandrojmp. It worked :slight_smile:

I think, I have missed the basic on this - to 0.0.0.0 instead of localhost . Same applies for UDP as well.

Summarizing the use-case here for the benefit of others:
Requirement:
Network Devices(UDP) --> AWS NLB --> FileBeat --> ES

Points to remember:

  1. AWS NLB/TG cannot do health check over UDP.
  2. Need to define a TCP input in Filebeat for this health check alone.

FB config below:
#------------------------------ Udp input --------------------------------
filebeat.inputs:

  • type: udp
    enabled: true
    host: "0.0.0.0:53"
    max_message_size: 10KiB

#------------------------------ TCP input --------------------------------

  • type: tcp
    enabled: true
    host: "0.0.0.0:9001"
    max_connections: 0
    timeout: 300s

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Checked the listen port in unix:
[root@d1entesttlsr001 ~]# netstat -tulpn | grep LISTEN
tcp6 0 0 :::9001 :::* LISTEN 1449/filebeat

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
** AWS NLB Health check settings:**
Protocol : TCP
Port : 9001
Status : healthy

Thank you once again.

Thanks,
Ravi

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.