Filebeat output configuration to TCP ports

Elastic Stack Beats
1

Hi,

Recently i started working on log forwarding to Kibana / ES and Apache NiFi thru logstash-forwarder and i am successfully finished the same. But, i came to know logstash-forwarder is deprecated and Filebeat is replacement of logstash-forwarder.

I configured logstash-forwarder with 50011 port which is enabled on ListenLumberjack processor inside NiFi. It reads all the logs and then it keeps them into ES and as well as into any DB. But, I am unable to forward the logs to any TCP (50011) ports from Filebeat. If so, i can stop using logstash-forwarder. Can someone please let me know is there any way to send the logs to Apache NiFi lumberjack with Filebeat ?

My current Flow :

Logstash-forwarder -> reads log files from system/apache -> sends -> NiFi ListenLumberjack (50011) -> Reads the logs -> Transforms the logs to -> Elasticseach & NiFi PutSQL processor.

Expecting Flow :

Filebeat -> reads log files from system/apache -> sends to -> NiFi ListenLumberjack (50011) -> Reads the logs -> Transforms the logs to -> Elasticseach & NiFi PutSQL processor.

Thanks in advance,

Regards,
Sid

2

Can you post your LSF and Filebeat configs? The NiFI ListenLumberjack implementation is something outside our control, so it could be that it's incompatible with Filebeat. OTOH, Filebeat uses pretty much the same protocol, so maybe it's just a configuration issue.

3

Please find my configs below :slight_smile:

LSF :

{
  "network": {
    "servers": [ "localhost:50011" ],
    "ssl certificate": "D:\\ELK\\Keys\\Test1\\lumberjack.crt",
    "ssl key": "D:\\ELK\\Keys\\Test1\\lumberjack.key",
    "ssl ca": "D:\\ELK\\Keys\\Test1\\lumberjack.crt",
    "timeout": 15
  },

  "files": [
    {
      "paths": [ "C:\\WINDOWS\\system32\\winevt\\Logs\\*" ],
	  "fields": { "type": "syslog" }
    },
	{
	  "paths": [ "C:\\Program Files\\Apache Software Foundation\\Tomcat 9.0\\logs\\localhost_access_log.*" ],
	  "fields": { "type": "apache-access" }
	}
  ]
}

filebeat:
  prospectors:
   -
    input_type: log
    paths:
      - C:\Program Files\Apache Software Foundation\Tomcat 9.0\logs\localhost_access_log.*
    document_type: apache-access

   - 
    input_type: log
    paths:
      - C:\WINDOWS\system32\winevt\Logs\*
    document_type: log

   -
    input_type: log   
    paths:
     - D:\lgs\*
    document_type: syslog

  registry_file: D:\ELK\filebeat\registry

output.logstash:
    hosts: ["localhost:5044"]
    bulk_max_size: 1024
output.file:
    path: D:\ELK\FB_Files
    filename: filebeat
    #rotate_every_kb: 10000
    #number_of_files: 7

shipper:

logging:
  files:
    rotateeverbytes: 10485760 # = 10MB
4

  1. you configured LFS with SSL enabled, but beats not (plain TCP).

  2. Skimming NiFi Lumberjack implementation, it seams to not support beats.

5

Thanks Steffens for your quick reply. I enabled SSL in beats and then tried invoking NiFi Lumberjack still i am unable to pass my logs to it.

Similarly the way we call output to logstash and file, can we pass it to TCP ? If so, can i use it like below ?

output.tcp:
hosts: [ "localhost:50011" ]

6

Did take some more time to read over the NiFi Lumberjack plugin. Unfortunately the implementation of NiFi only supports LSF, but not beats.

Similarly the way we call output to logstash and file, can we pass it to TCP ? If so, can i use it like below ?

output.tcp:
hosts: [ "localhost:50011" ]

There is no output.tcp module in beats. TCP is missing on purpose, as with plain TCP we don't get any kind of good ACK telling us how far the receiving application has successfully consumed the lines published by filebeat.

7

Yeah even no NiFi processors are reading the logs directly and then we can keep it into any other processors.

is there any workaround for this ? If so please suggest me.

Thanks,
Sid

8

I have no idea about NiFi.

What's wrong with GetFile processor? Check the docs https://nifi.apache.org/docs.html.

Using filebeat with NiFi, maybe you can leverage Kafka or Redis.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.