SPLIT Pipeline is not working correctly

Elastic Stack Elasticsearch
1

Hello All,

Thanks in advance.

We have configured the logstash on ServerA and pipeline is configured. All the filebeat agents are sending data successfully to ServerA.
PipelinePrimary.conf

To avoid delay and better load management. We have implemented a new logstash server ServerB and configured separate pipeline.conf file .
Pipeline1.conf
Pipeline2.conf

ServerA PrimaryPipeline.conf has port 3101
ServerB Pipeline1.conf has port 3102
ServerB Pipeline2.conf has port 3103
Each pipeline have different ports

==============================================================================================================================
On first agent server server1 we have 2 filebeat.yml file. Filebeat1.yml pointing to ServerA logstash and utilizing the grokking defined on Pipelineprimary.conf. via port 3101
On same agent server server1 the second filebeat, Filebeat2.yml pointing to SerevrB Logstash and utilizing the grokking defined on Pipeline2.conf.via port 3103

Both the filebeat.yml files on server ServerA is successfully sending the data.

==================================================================================
On second agent server Server2 we have 3 filebeat.yml file.

On Server2 Filebeat1.yml pointing to ServerA logstash and utilizing the grokking defined on Pipelineprimary.conf. via port 3101

On same agent server Nodeserver2 the second filebeat, Filebeat2.yml pointing to SerevrB Logstash and utilizing the grokking defined on Pipeline1.conf.via port 3102
On same agent server Nodeserver2 the third filebeatc, Filebeat3.yml pointing to SerevrB Logstash and utilizing the grokking defined on Pipeline2.conf.via port 3103

Successfully sending data via primarypipeline (via port 3101) and Pipeline1.conf (via port 3102)
Unfortunately on NodeServer2. No data is getting send for the logs defined in the filebeat3.yml which is utilizing the grokking defined on Pipeline2.conf for server ServerB.(port 3103)

on servers where two pipelines are configured via port 3101 and 3102 the data is sending successfully.

We have noticed on servers where three pipelines are configured the third Filebeat3.yml is not sending any data to pipeline2.conf via port 3103. Other two filebeat.yml are sending data.

Can any one help on this.

2

It is a little confusing.

When you say 3 filebeat.yml you are talking about 3 different filebeat instances?

This is not clear, you have multiple filebeats running on the server or just one filebeat?

Please share all the filebeat.yml files specifying fomr which instance/server this filebeat is, also share your Logstash configurations, or at least the inputs and outputs.

3

yes on server NodeServer2 we have 3 different filebeat instances.

4

In short

Server where two filebeat instances are running sending data to Logstash server.

Server where three filebeat instances are running the third filebeat instance not sending any data to the Logstash server.

5

You need to share the configurations, it is not possible to know what may be the issue without it.

Also, what do you have in the logs of the instance that is not working? Double check if all filebeat instances are correctly configured to use different data paths for the registry file for example.

6

Logstash is running on the windows server.
The same pipeline.conf is working on one server log on local disk.
The same pipeline.conf is not working on another server log on local disk.

Log having similar path. Filebeat instance are properly configured.

The issue is happening on server where the split pipeline coexist.

7

I can't provide any further help without seeing the filebeat.yml files you are using and the logstash configurations.

8

Thanks for your response.
Pipeline just contains an input filter and specific filtering and output.
Filebeat contains the log location, port and destination logstash server details nothing more.
The issue happening on servers where three filebeat services coexist . Is there any specific configuration needs to follow if the multiple file beat instances on same server reporting to same logstash server via different pipeline(split pipeline).
The filebeat instance are successfully sending the data if its utilizing only one of the pipeline.conf (Split pipeline method).

Each pipeline have different ports

9

As I said, I can't tell without seeing the filebeat.yml you are using and your logstash pipeline configurations, it is also not clear what you mean by split pipeline.

You need to share your configurations if you want further advice, without it is not possible to help.

10

Split pipeline means Multi Pipeline. Logstash pipeline.conf contains more than 1000 of lines.

11

Are you using pipelines.yml to have multiple pipelines or you just have one big pipeline config with conditionals?

If you have conditionals, check your conditionals.

As I said, I can not help further without seeing the configurations you are using.

12

Multiple pipelines with different conditionals and different port numbers. Verified the conditionals already same conditional is working for other servers. The issue exist where 3 filebeat services are running and utilizing multiple pipeline.conf for sending data to the logstash. The pipeline.conf contains more than 1000 lines. So very difficult to copy paste over here.

13

Filebeat uses filebeat.yml, how your logstash configuration looks like doesn't matter for filebeat as it has no knowledge about it, logstash is just an output.

You need to share your filebeat.yml for the instances on the server that are not working, if you can't share it here try using pastebin for example, but without seeing the configurations is pretty hard to guess what may be the issue.

14

For your information. The same filebeat.yml is working on another server. All the log files are on local disk so the path is same. The only difference on working and non working is server name on file beat.

below is the filebeat.yml configuration

type: log
enabled: true
paths:
E:\Program Files\clone*.dat
encoding: utf-8
fields:
FileBeatLogType : TYPE24
SubApp : Blockpdf
fields_under_root: true
ignore_older: 356h
multiline.pattern: '^#'%@'
multiline.negate: true
multiline.match: after
multiline.max_lines : 40000

name: Server2

fields:
Environment: Prod
Application: Argconick

fields_under_root: true

max_procs: 8

output.logstash:

Boolean flag to enable or disable the output module.

enabled: true

The Logstash hosts

hosts: ["loadbalanc.midligh.com:3102"]

The default value is 3.

compression_level: 3

Optional index name. The default index name is set to filebeat

in all lowercase.

index: 'filebeat'

#================================ Logging ======================================

Configure logging | Filebeat Reference [8.8] | Elastic

logging.to_files: true
logging.level: info
logging.files:
path: C:\elk\filebeat_3102\Logs
name: filebeat
rotateeverybytes : 20971520
keepfiles : 10
rotateonstartup: true

Also, the total values for all non-zero internal metrics are logged on shutdown. The default is true.

logging.metrics:
enabled: true

The period after which to log the internal metrics. The default is 30s

--> We take a higher value so that the log isn't growing to rapidely.

period: 300s

15

So, is the server that is not working able to talk with Logstash? This is the first thing you need to check.

Start the filebeat on this server and check the logs.

16

Always we are receiving the below error on non working filebeat server.

========================================

beater/filebeat.go:178 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled.
If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.

17

You will have other log lines, please share them when you start filebeat.

Also, use the preformatted text option, the </> button when sharing log lines and configuration.

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.