Filebeat-logstash question

Elastic Stack Beats
1 
filebeat.inputs:
  - type: filestream
    id: tad-run-logs
    enabled: true
    paths:
      - /run/*/*.tl
      - /run/*/extra_info
    exclude_files:
      - '/run/cleanup_rack/'
    file_identity.native: ~
    prospector.scanner.check_interval: 1s
    backoff.init: 1s
    backoff.max: 2s
    backoff_factor: 1
output.logstash:
  hosts: ["log02:5044"]
  timeout: 120s
  workers: 1
  pipelining: 0

logstash is simple basically just parsing file path and extension

output {
  file {
    path => "/s1/log/sachin/%{job}.%{extension}"
    flush_interval => 1
    stale_cleanup_interval => 30
    codec => line { format => "%{message}" }
  }

I am starting logstash manually to test whole setup

/usr/share/logstash/bin/logstash --pipeline.batch.size 4096 --pipeline.ordered true --pipeline.workers 1 --pipeline.batch.delay 50  -f /etc/logstash/conf.d/my_file.conf

it works, sends data and it gets in to own file, line by line which is why I have pipeline worrkers 1 and pipeline.ordered=true

but it sends exactly 1600 lines every few seconds nothing more nothing less untill it reaches to end of file

logstash output is something like these where it open's file, close every min and then opens right away.

[INFO ] 2026-05-12 11:56:49.599 [[main]>worker0] file - Opening file {:path=>"/s1/log/sachin/283213924.extra_info"}
[INFO ] 2026-05-12 11:56:49.601 [[main]>worker0] file - Opening file {:path=>"/s1/log/sachin/283213924.tl"}
[INFO ] 2026-05-12 11:57:35.622 [[main]>worker0] file - Closing file /s1/log/sachin/283213924.extra_info
[INFO ] 2026-05-12 11:57:35.622 [[main]>worker0] file - Closing file /s1/log/sachin/283213924.tl
[INFO ] 2026-05-12 11:57:51.249 [[main]>worker0] file - Opening file {:path=>"/s1/log/sachin/283213924.extra_info"}
[INFO ] 2026-05-12 11:57:51.251 [[main]>worker0] file - Opening file {:path=>"/s1/log/sachin/283213924.tl"}
[INFO ] 2026-05-12 11:58:40.615 [[main]>worker0] file - Closing file /s1/log/sachin/283213924.extra_info
[INFO ] 2026-05-12 11:58:40.615 [[main]>worker0] file - Closing file /s1/log/sachin/283213924.tl
[INFO ] 2026-05-12 11:58:53.117 [[main]>worker0] file - Opening file {:path=>"/s1/log/sachin/283213924.tl"}
[INFO ] 2026-05-12 11:58:53.118 [[main]>worker0] file - Opening file {:path=>"/s1/log/sachin/283213924.extra_info"}
[INFO ] 2026-05-12 11:59:45.653 [[main]>worker0] file - Closing file /s1/log/sachin/283213924.tl
[INFO ] 2026-05-12 11:59:45.654 [[main]>worker0] file - Closing file /s1/log/sachin/283213924.extra_info
[INFO ] 2026-05-12 11:59:54.309 [[main]>worker0] file - Opening file {:path=>"/s1/log/sachin/283213924.tl"}
[INFO ] 2026-05-12 11:59:54.310 [[main]>worker0] file - Opening file {:path=>"/s1/log/sachin/283213924.extra_info"}

my goal is to have this running on system where multiple file gets generated by running process and then removed. I want to send them to logstash server and put it in common location. basically like common log place.

how do I send more log line. and or when more log line comes logstash should handle it faster. I have enough resource(memory/cpu) on logstash system.

2

I change this setting on logstash startup

--pipeline.batch.size 1

now in startup whole file gets sync in one go. but then subsequent sync does not work fast enough. it gets only few new lines every few second. and file open/close becomes too frequent

INFO ] 2026-05-12 13:00:01.189 [[main]>worker0] file - Closing file /s1/log/sachin/283213924.tl
[INFO ] 2026-05-12 13:00:36.179 [[main]>worker0] file - Closing file /s1/log/sachin/283213924.extra_info
[INFO ] 2026-05-12 13:00:51.784 [[main]>worker0] file - Opening file {:path=>"/s1/log/sachin/283213924.extra_info"}
[INFO ] 2026-05-12 13:01:16.192 [[main]>worker0] file - Closing file /s1/log/sachin/283213924.extra_info
[INFO ] 2026-05-12 13:01:27.789 [[main]>worker0] file - Opening file {:path=>"/s1/log/sachin/283213924.extra_info"}
[INFO ] 2026-05-12 13:02:16.188 [[main]>worker0] file - Closing file /s1/log/sachin/283213924.extra_info
[INFO ] 2026-05-12 13:02:29.793 [[main]>worker0] file - Opening file {:path=>"/s1/log/sachin/283213924.extra_info"}
[INFO ] 2026-05-12 13:03:41.200 [[main]>worker0] file - Closing file /s1/log/sachin/283213924.extra_info
[INFO ] 2026-05-12 13:03:41.802 [[main]>worker0] file - Opening file {:path=>"/s1/log/sachin/283213924.extra_info"}
[INFO ] 2026-05-12 13:06:26.181 [[main]>worker0] file - Closing file /s1/log/sachin/283213924.extra_info
[INFO ] 2026-05-12 13:06:27.834 [[main]>worker0] file - Opening file {:path=>"/s1/log/sachin/283213924.extra_info"}
[INFO ] 2026-05-12 13:07:23.770 [[main]>worker0] file - Opening file {:path=>"/s1/log/sachin/283213924.tl"}

3

I'm not sure exactly what is the problem you are trying to solve and the issue you are having.

What version are you using for both Filebeat and Logstash?

What are you trying to control with those batch size?

4

issue is that filebeat slows down sending data or logstash is not putting data fast.

in initial startup it does sends all thousands of line in few seconds ( this tells me there is no resource problem or network issue on both side), but then it just comes in batches. hence I am trying to figure out which setting that controls that and what would be good for my use case.

5

What are the versions for both filebeat and logstash?

They work in batches, what you can do is try to change to batch sizes to see if it reaches your desired ingestion rate.

You seem to not have changed the Filebeat bulk_max_size, in later versions this value has a default of 2048, and your Logstash batch size is 4096.

Both are maximum value, filebeat and logstash may emmit smaller batches, larger batches can also impacting in speed processing.

You also have 1 worker on both filebeat and Logstash, this will impact your ingestion if you have multiple files that are being written at the same time.

Try to set the batch size for Logstash the same as the one in Filebeat (2048).

6

Testing out 8.19.15 filebeat. Logstash in other hand is 8.16.1. do I need both same? I can change them as these is POC.

I am testing pipeline.batch.size to 50 now. smaller chunk. because filebeat side has either two or four file that needs to send data. and they are not in large volume. Eventually source server will be lot more about 50 servers and each will have two - four files which can have few line appended ever few second and sometime nothing for hours.

Basically trying to create central log location for these files.

7

The recomendation is to use the same version, 8.16.1 is not supported anymore, I would use both in the last 8.19.* version.

From your description it seems that you do not have a problem as this is still a POC.

Do you have a requirement to have things indexed in a ordered way when you can order it by using the offset field from the file?

I would suggest that you start with the defaults and only worry with tunning logstash/beats if you face some issue.